The pump should work OK in the engine bay, but you might want to put it away from the engine heat. One possible source of the bubbles you saw would be the fuel boiling in the lines.
Another possibility would be a leak somewhere, allowing the pump to suck air, possibly combined with a restriction in the line. You might try removing the inlet line from the pump and blowing backwards into the tank; then inspect inside the tank to see if anything came out or is floating around back there.
I can't explain the fuel pouring out the vent, unless perhaps the tank was just overfilled. Because of the vent location, you can't fill it until the nozzle shuts off. I actually blocked mine off permanently, since even with some air space in the top of the tank, a hard left turn will dump fuel out the vent (as the fuel sloshes to the side with the vent tube). My car already had a vent hole in the cap, but it would be easy enough to add if your car doesn't have it.
