I ran into this problem when installing Untangle the other day and I have seen quite a few other people who have run into this problem.  In a note to myself, and hopefully help some other people out along the way, here is the solution.

I was installing Untangle on a Dell machine with on-board video which is setup with a 1MB video buffer in shared memory and when the XDM probed the video card I had some errors to the effect it could not probe the display device correctly.  The XWindow session started, but the screen was larger than the display resolution which meant I had to move windows around to even be able to view the window content. 

I did some digging around in the Untangle forums and came across a solution.  This works with Untangle, but it should also work with other distributions of Linux if you are having problems probing your video card.  In the case of the Dell machine I was loading it defaults to 1MB shared video memory for the video card.  Changing the shared video memory from 1MB to 8MB fixed the probing errors and the problem was resolved.  No more beating my head on a desk trying to figure out the problem.