Born in Chichester - at that time Sussex, now West Sussex, James Hayllar begun his studies at the Cary's Art Academy in London (in 1851 he painted a portrait of Francis Steven Cary rector of the Academy and already famous painting teacher of some Pre-Raphaelites such as Millais and Rossetti) and then attended the Royal Academy.