Why is this?
Firstly, Highway Engineering tends not to attract intellectual high flyers. The clever, dynamic engineers tend to go into building structures, bridges etc at the more glamorous end of the industry. Highway Engineers are therefore a little intimidated by well informed cyclists, which is why they don't come and talk to me, and why they hide behind obscure regulations when their designs are challenged. They also don't always know their subject very well.
The Highways Agency "Design Manual for Roads and Bridges" is a huge, Byzantine collection of disconnected advice notes. Knowing all the regulations affecting any given situation is well nigh impossible. It is almost universally inapplicable to the sort of situations where cycle infrastructure is needed (it is aimed at National Roads, Trunk Roads and motorways - not local roads) but engineers use it anyway because it is the only prescriptive guidance available. The sprawling nature of the regulations also makes it very difficult for lay people - campaigners, councillors, individuals - to challenge designs. The designer only has to claim it is in accordance with such and such regulation and it will get rubber-stamped. Whether that regulation is correct or relevant will probably never be checked.
Highway Engineers all seem to drive to work, so their impressions of cyclists come through the windscreen - cyclists are usually a "them", not and "us", and all the usual prejudices apply. In my 20 years of working I have noticed that cycle sheds are populated mostly by structural engineers, architects and geographers - all creative thinkers, with very few highway engineers. This is a reflection in the highway engineering design culture - applying prescriptive rules dictated by others, and doing what everybody else does rather than thinking for themselves.
Neither Highway Authorities (County Councils - publicly answerable), nor their designers like "risk". We are of course talking about the risk of being sued, not risk to members of the public. They therefore stick to prescriptive guidance, whether it is relevant or helpful. If a DfT advice note says make cyclists jump off a cliff then that is what they will do, and the Safety Audit will endorse it. If there isn't an advice note detailing how to do it, it will never happen.
Nor do they like criticism. Of course they get a lot of criticism from cycling groups but cyclists are a "them" and cyclists' views, however accurate and clearly argued, can therefore be written off as extreme, politically motivated, single issue, or just the ramblings of a bunch of tree-hugging, weirdy beardies. Criticism that counts comes from the newspapers, and from friends and colleagues in the car park (they are of course an "us" in us / them terms). It is extraordinary to see how peoples' attitudes suddenly switch where cycling issues come into a conversation. One minute I am a highly respected designer, entrusted with a multi-million pound project, then suddenly I am a self-interested, single issue trouble maker who knows nothing - all because we strayed into territory touching on the other person's attitudes and behaviour. People know that driving causes all manner of ills, but they can't handle the feelings of guilt that come with acknowledging that, so they pretend it is all somebody else's fault - him over there, that ***** in the lycra.
As for the easy life - making space for good quality cycle infrastructure is never easy. It takes drive, determination and the will to stand up and take criticism from all sides. The path of least resistance is the nasty compromises we see all over Britain - no use to cyclists, but fulfilling the political imperative of being seen to be green.