The BLS is a branch of the Dept. of Labor. And the Dept. of Labor is headed by the Secretary of Labor who was appointed by Obama.
BLS is administratively part of the DOL, but DOL has no input into the management or running of any of the programs. The Commissioner of BLS was appointed by Bush. No one outside of the program office has access to any of the data prior to release and the offices are locked down with no entry to those not assigned to that office without escort. The night before the release the Sec Labor and the President are given the initial release. Where in that process, and by whom, is the manipulation supposed to occur?
In the case of the CPI, there is a massive amount of money at stake. A lower CPI benefits government and corporations while reducing the standard-of-living of ordinary citizens.
All of which, and your other points about the CPI-U are true. But that does not mean that the career economists/statisticians are not neutral...they have no incentive or reason to manipulate especially when they are forbidden from trading in any products directly tied to the CPI. The only person in BLS who is politically appointed is the Commissioner. At best he can influence broad methodology.
Your last question is more complex and too lengthy to address here, but I'll mention three changes implemented in the 90s after a lot of beating of drums by Alan Greenspan and his ilk:
1) calculating housing costs by "equivalent rent"
2) hedonic adjustments
3) substitution/geometric weighting
Ok..all three are controversial, but my question was how they understate inflation. But let's take a quick look anyway.
1) The purpose of the CPI is to look at Consumption. Housing has a strong investment component. Therefore, to negate the investment portion, a rental equivalent is used to divorce the consumption part from the investment part. That's the official reason. If you're claiming the real reason is to understate and/or manipulate, then please show how the the official explanation is not true. Merely pointing out or claiming a lower index by the method does not mean that that the concept is dishonest.
2)hedonic adjustments. Items change over time and to maintain continuity you have to adjust for quality changes. Hedonic adjustments result in both higher and lower changes than they would without adjustments. Hedonic adjustments are certainly not perfect...computers are a pain in the ass to adjust, and there's legitimate arguments about precisely how to do it and what components and what weights are appropriate. I'm not aware of any evidence to suggest that proper adjustment is not the real purpose or any way in which the numbers are specifically manipulated.
3)substitution/geometric weighting. A Laspeyres index assumes no substitution and so gives the maximum price change possible. But substitution does occur and to ignore that will OVERSTATE the actual change in prices. Geometric means indexes are used only at the base level for the specific area (bread in Los Angeles, for example) and not hamburger for steak as many detractors falsely claim. At the higher index level, a Laspeyres index is still used. Research suggests the differences since the change to geometric means has been minimal. Again, where's the evidence that the intent is manipulation or that the concept is dishonest?