A breakaway group of around 50 Somali militants who declared loyalty to the Islamic State jihadists abandoned a small port town on Thursday a day after occupying it, residents and an official said.