Born in Tokyo, the son of a man of property, Matashichi Toriyama, in Shinagawa. He was adopted by his uncle, who was the head of the Hirabayashi Family. In 1851, he took over as the head of the family. After the Meiji Restoration, he served as chief representative of the ward, and a member of the Tokyo Prefecture assembly. In 1892, he was elected to the House of Representatives. While actively working as a statesman, he devoted himself to the development of his local region using his personal funds. He was awarded the Medal with Blue Ribbon in 1900 in recognition of his services.