Why exactly does the carbon hydrogen bond length decrease from 1.099 to 1.070 angstrom in the series ethane, ethylene to acetylene? According to common wisdom it has to do with orbital hybridization. As the hybridization changes from sp3 to sp2 to sp in the carbon atoms of those molecules, the s-character increases and the p character decreases. Because of the particular shapes of s and p orbitals meaningful s-s orbital overlap requires shorter bond lengths compared to the corresponding p-p orbital overlap. In a recent article Vermeeren, Bickelhaupt and Fonseca Guerra want us to change this view (DOI). Based on theoretical calculations they do not look at hybridization but rather at Pauli exclusion as the main cause.
For the explanation you may want to sit down. Before they deal with Pauli exclusion they first eliminate strain energy as another potential driver. After all, compared to the two hydrogens in acetylene, the 6 hydrogen atoms in ethane tend to crowd each other and a longer carbon carbon bonds surely helps to keep them at a distance. The bond-dissociation energy can be taken as the sum of the strain energy and the so called internal energy but the authors point out that in DFT calculations the strain energy is just a fraction of the larger internal energy.
The internal energy can also be broken down into components parts. They are first of all an electrostatic interaction. The orbital interaction energy is the result of the formation of a chemical bond when two molecular orbitals approach with each a single electron, a.k.a. a SOMO. Imagine two methyl radicals approaching each other to form ethane. Both parts contribute positively to bond formation. That leaves the third part, the Pauli repulsion.
Is this a vague term? Wikipedia does not think it warrants a dedicated page and instead it is mentioned in the context of a concept called exchange interaction. It also gets thrown in the same box as steric repulsion. It is certainly related to the Pauli exclusion principle which states that it is impossible for two electrons of a poly-electron atom to have the same values of the four quantum numbers. In any event, the repulsion whatever you call it, prevents two atoms occupying the same space and at short bond lengths it is always a destabilizing force.
Then which of the three components is driving the bond lengths? The article first describes where on the carbon to hydrogen axis the orbital overlap reaches a maximum. It turns out this maximum is situated at much shorter distances than the actual bond lengths with much larger variation in values as well. This pretty much rules out orbital interaction energy as a main driver. In a second plot, this time of bond length against bond energy Pauli clenches the deal as it is the only component that actually decreases in the order ethane > ethylene > acetylene.
The only thing to do next is substitute Pauli repulsion with coordination number and all the chemistry textbooks can be rewritten, the authors suggest in conclusion. I did not check all my now obsolete textbooks (I can throw out my copy of Jolly's Modern Inorganic Chemistry and Advanced Organic Chemistry by Carey/Sundberg all from 1985) but at least Jerry March dodges the bullet. In his Advanced Organic Chemistry (again 1985) He writes "however, other explanations have also been offered and the matter is not completely settled". Or should he have written "and more explanations will be offered in the future"?