Differential Scope Primer is a good introduction to oscillospe measurement. This web posting is a shortened version of the information on that document related to ground loops. I have also added here some of my own comments.
Most oscilloscopes are designed to measure voltages that are referenced to earth ground, which is connected to the scope chassis. These are referred to as “single-ended” measurements. Ground loops can corrupt such scope measurement easily.
A ground loop results when two or more separate ground paths are tied together at two or more points. The result is a loop of conductor. Connecting the ground lead of an oscilloscope probe to the ground in the circuit-under-test results in a ground loop if the circuit is “grounded” to earth ground. Typically the metal chassis of both scope and device under test are connected to safety ground and internal power supply common. Scope probe ground connects to scope chassis at the input BNC connector.

Image source: http://www.tek.com/Measurement/App_Notes/DiffOscPrimer/Introduction.html
In the presence of a varying magnetic field, this loop becomes the secondary of a transformer which is essentially a shorted turn. The magnetic field which excites the transformer can be created by any conductor in the vicinity which is carrying AC or changing current. The potential difference seen on oscilloscope probe ground can range from microvolts to as high as hundreds of millivolts.
In these situations, it’s often tempting to remove the probe ground lead to get rid of the noise. This method really works sometimes, but this technique is only effective when measuring very low-frequency signals. At higher frequencies, the probe without good ground contact on both ends begins to add “ring” to the signal. Keep in mind that the current must always form a loop and the smaller the loop are the better. Mysterious ground article gives you more details on what happens and what errors you can see. It is possible that the position of the probe cable can have an effect on the shape of the signals you see on the scope (Try it). Another nasty artifact of a no-ground probe arrangement is the resonance associated with the combination of the rather large inductance (loop inductance of L1=500 nH) and input capacitance of the probe (for example C1=1-10 pF). This resonance is called a probe resonance. A short, explicit ground connection made between the scope ground and the equipment under test shunts around both CI and L1, eliminating their influence on the measured result and pushing the probe resonance up and out of the band of interest.
The next technique often tried to break ground loops is to “float” the scope or “float” the circuit being measured. This practice is inherently dangerous, as it defeats the protection from electrical shock. Idea of “floating” the scope is generally a bad and unsafe idea with a normal oscilloscope (usually metal case and touchable metal parts in it, all in contact with probe ground). Some battery-operated portable scopes allow safe floating operation and you can get rid of ground loop problems and neither side grounded problems with them.
In case of small circuit being measured powering the circuit through safety isolation transformer that breaks the ground connection could be useful. Powering the circuit being measurede though safety isolation transformer is a proven method used at electronics repair shops.
Even when the measurement system doesn’t introduce ground loops, the “ground is not ground” syndrome may exist within the device being measured. Large static currents and high-frequency currents act on the resistive and inductive components of the device ground path to produce voltage gradients. These effects have challenged designers of sensitive analog systems and fast digital systems for years.
If the voltage to be measured is between two circuit nodes, neither of which is grounded, conventional oscilloscope probing cannot be used. There are several types of differential amplifiers and isolation systems available for oscilloscopes with different propertied (targeted for different applications).