THE COUNT OF THE SAXON SHORE

[Nathan Ross]

it doesnt look to me like the remaining population would have been enough to defend the walls if the occasion arose...

Somebody appears to have been defending something in the later 4th or early 5th century, although as usual the scrappy evidence isn't giving us many clues about what was really going on!

"The gate appears to have been burnt on at least two occasions in the late Roman period. The second fire was the more serious. Brushwood was piled up against the outer face of the wooden doors and set alight causing the doors to collapse inwards. The heat was so intense that many stones turned red. The two fires are not closely datable but the first was probably no earlier than 367 and the second substantially later. The fact that there were two fires shows that the town survived the first assault (if indeed that was the cause of the first fire). The debris from the second fire, however, does not appear to have been cleared away, implying that the gate was never repaired." 

"The fate of the Romano-British population of Colchester is unclear but life in the town was certainly radically different by the mid 5th century, the date of the earliest known Saxon hut... Before the change, there is likely to have been a period when the population was in rapid decline."

Iron Age and Roman Colchester (1994)

I don't know if there has been much new work done on the later period there since.

As for the Comes Litoris Saxonici, my own favoured theory is that the commands of both he and the Dux Britanniarum as listed in the ND were set up c.370 and replaced whatever previous arrangements might have been in place, including the wall garrison, which survives as one of many anachronisms in the ND's list. As with most other things in this period, these commands were in a state of constant flux and probably did not survive in that form until the end of the 4th century.

By the early 5th century, I think, most troops in Britain would have been settled Germanic foederati and local militia, with only the mobile field command of the Comes Britanniae acting as a strategic reserve - the latter were probably billeted in towns or even larger villas across southeastern Britain, until c.AD407.

The Comes Litoris may have been based at Dover, or have moved his headquarters around the country. He may even have established himself at one of the surviving big villas of the coastal region - Eccles and Darenth in Kent, for example, are usually believed to have been downgraded to 'industrial use' by this period, but may have been taken over by the military instead, perhaps?