The Hall was originally built in the 19th Century by the Astley family who have been very successful in the tea trade. Local tradition has it that the house was built in the style of a tea caddy but it is much more likely that the house was built in the colonial style, familiar to tea planters and this, together with the rather square style of the original building, gave rise to the story.
In or around 1906 the property was purchased by Mr William Lancaster JP, a prosperous businessman, who renovated the property. One of his actions was to have the gates of the old King's Lynn slaughter house, "The Shambles", removed to the Hall when it was closed down in 1912.
William Lancaster used his large fortune to benefit the locality including King Edward VII Grammar School for which he was rewarded a Knighthood.
One of his grandchildren was Osbert Lancaster, now Sir Orbert, the famous author and cartoonist who recalls childhood visits to the hall in his book "All done from memory". In the books he refers to terra-cotta masks of Comedy and Tragedy enlivening the plain expanse of yellow brick between the first and second storey. The masks and the yellow brick were lost following a fire around 1960 which necessitated some rebuilding with non-matching bricks so the whole building was whitewashed to disguise the difference.
After the death of Sir William in 1828 the Hall was held in trusteeship by Lionel Rodwell who added the East Wing, and then through succession of owners including Mrs Hume-Gore, Mr Cragg, Mr Wilson and eventually to the Vassalos family who owned the Campbell's Soups factory in Lynn and used the Hall for entertaining.
In 1970 the freehold was purchased by the Dredging and Construction Company who completely changed the building by adding the three-floor office block to the rear. The Dredging and Construction Company occupied the offices until the end of 1980 when they were purchased by the Hoechst as a Head Office and Farm site for its agricultural Division.
In December 2000 it became home to a new business - Adrian Flux Insurance Services.