The affair between Henrietta Howard and George, Prince of Wales (the future George II) may have begun at Richmond Lodge, probably in 1718, (Tracy Borman: King’s Mistress, Queen’s Servant). The proximity of Richmond Lodge may also have been a factor in Henrietta’s decision to purchase land in 1724 in Twickenham to build Marble Hill. However, Tracy Borman suggests in her biography of Henrietta that the principal reason why she built her house there, was probably because it would have been near to her close friend, Alexander Pope, who also had recently built a villa at Twickenham. The Prince of Wales had taken a lease of Richmond Lodge, because of his obsession with hunting. Henrietta Howard said: “We hunt with great violence and every day have a tolerable chance of our neck being broken”. Richmond Lodge was too small for all the Prince of Wales’s retinue and he rented a row of houses in 1724, which had been built in 1717 as a speculation, for his Maids of Honour, and the Row became known as Maids of Honour Row (Sally Jeffery: The Building of Maids of Honour Row, Richmond: Georgian Group Journal: 2010). Henrietta never lived there. As a Woman of the Bedchamber, she had to be on hand to wait on Princess, later Queen Caroline.
Richmond Lodge was at the south-west corner of the present Royal Botanical Gardens. The lodge was probably built in the early 17th century as a hunting lodge for James I, but was extensively altered and remodelled for William III again in the late 17th and early 18th centuries. It is possible that Nicholas Hawksmoor may have been involved in the rebuilding of Richmond Lodge. William III used it himself as a hunting lodge, then let it to a friend John Latten who assigned the lease to the Duke of Ormonde, who was a distinguished soldier, but he had to go into exile for supporting the 1715 Jacobite Rebellion. His estates were confiscated, apart from Richmond Lodge, which was saved thanks to his brother Lord Arran. In 1718 the Prince of Wales took the lease of Richmond Lodge from Lord Arran.
However, in 1727 when George II became King, his wife Queen Caroline, decided to create one of the first English informal gardens. The gardens extended from Kew all the way to Richmond Green. She commissioned Charles Bridgeman to landscape the gardens. He built a terrace walk along the Thames to Kew and a canal and a “wilderness” within the grounds. Caroline also commissioned William Kent to build two extraordinary follies, The Hermitage and Merlin’s cave. The former had busts of philosophers and the latter had waxworks with a poet, Stephen Duck, as its resident custodian. Queen Caroline died in 1737 and no more work was done on Richmond Lodge until 1760 when George III inherited the lodge on the death of his grandfather, George II.
After George III married Queen Charlotte in 1761, they used the lodge as a country retreat. He decided to have the whole park landscaped once again, commissioning Capability Brown. The “landscaping” included pulling down the hamlet of West Shene. This hamlet had faced the Thames at the south-west corner of the estate. George III also instructed Sir William Chambers to build an observatory in the grounds to observe the transit of Venus in 1769. (The observatory is still there). George III was planning to replace Richmond Lodge with a new palace. However in 1772, he inherited the White House, opposite the present Kew Palace, from his mother. George III decided to make the White House his country retreat in place of Richmond Lodge. George III had recently bought Buckingham House (now Buckingham Palace), so he had Richmond Lodge pulled down, but could not afford to replace it, as he had limited funds available . The White House itself was pulled down in 1802.