Bandits sometimes appear on your islands if you've either refused to let in or kicked out Beggars from your cities. Bandits will encamp quickly and beginning seizing your island's structures within their area of influence, preventing these structures from producing any goods or services.
To get rid of them, you can either pay them money which converts them back into Beggars or you can destroy them militarily. The cost to pay them depends on the size of the Bandit encampment. Bandit camp strength is sufficient to easily take down a Watchtower or an Oriental encampment, so be sure to use enough force. Also have at least 1 Large military camp support other camps when attacked or let all simultaneously attack the bandit camp.
There are three different sizes of Bandit encampments: