If you are considering solids, the molecules are held into rigid frameworks. There are a number of ways to create rigid frameworks of molecules in which the average distance between molecules was larger than the actual distance between any two molecules. This is done by structural limiting the total number of molecules that any one molecule can interact with. For example, suppose the molecules want to interact at a distance of 3 Å. In a liquid, a molecule might have as many as twelve neighbors all at the same distance of 3 Å. If you put that molecule into a lattice at the vertex of a cube, however, you reduce that number to 6 nearest neighbors, and the average distance between molecules increases as the number of molecules at that distance decreases - the solid is less dense than the liquid. If you add voids to the crystal structure, as in foams or zeolites, you can get further reductions in densities of the solids.
You do not have these rigid arrangements in gases and liquids. In fluids, the density is fixed by the preferred distance between the molecules, and the total number of molecules at that distance is determined only by the volume the material is occupying. In gases, there is no preferred distance between the molecules - the molecules will disperse to fill as much space as possible. Increase the volume, density decreases. Liquids do have a preferred distance between molecules - as you increase the volume of the container, the volume of the liquid stays constant. Increasing pressure on a liquid does not change the distance between molecules very much, while increasing pressure on a gas will change the distance between molecules until the energetics of condensing to form a liquid become favorable.Then the liquid will condense, leaving the gas with more space to move around.