How’d I Do on My 2015 Mass Atrocity Forecasts?

22 Jan

For the third year running, I took a stab at predicting which countries would experience mass atrocities in 2015 (defined as 1,000 noncombatant intentional deaths caused by a discrete group in a calendar year) last January. Here’s what I predicted. I’ve put “YES” next to places that did experience atrocities and “NO” next to the countries that didn’t. For the countries where I’m not willing to hazard a guess based on insufficient data, I’ve put a “?”.

  • Nigeria (95%) YES
  • Iraq (95%) YES
  • Syria (95%) YES
  • Pakistan (75%) NO
  • Afghanistan (70%) YES
  • South Sudan (70%) YES
  • Sudan (65%) NO
  • Mexico (55%) ?
  • CAR (50%) NO
  • North Korea (50%) ?
  • Somalia (30%) NO
  • DRC (30%) NO
  • Libya (25%) NO
  • Gaza (25%) NO
  • Cameroon (20%) NO
  • Ukraine (10%) NO
  • Rwanda (10%) NO
  • Lebanon (10%) NO
  • Burundi (5%) NO
  • Yemen (5%) YES
  • Chad (5%) NO
  • Guinea (5%) NO
  • Kenya (5%) NO
  • Ethiopia (5%) NO
  • Burma (5%) NO
  • Eritrea (5%) NO
  • Zimbabwe (5%) NO
  • Mali (5%) NO

Going forward, if you’re interested in looking at the numbers or the analysis, then read the whole thing. If you’re just interested in basic conclusions, read only the MAIN TAKEAWAY portions.

One method to figure out how successful I was is to see each case for which I put forward a prediction as containing 100 points. If an atrocity happened, I get the percentage I predicted that an atrocity would happened, and if no atrocity happened, then I get the result of that percentage subtracted from 100. For example, I’ll get 95 points for Burundi but only 5 for Yemen.

Using this method, I get 2035 out of a possible 2600 (this excludes Mexico and North Korea for which I couldn’t make a judgement.) Initially that sounds pretty good, coming in at 78% accuracy, while in 2014 I was 68% accurate. However, my numbers are of course padded by the high probability countries and the low probability countries. If I only look at countries between 90%-10%, I’m 68% accurate, whereas if you look between 80%-20%, I’m only 62% accurate. Regardless, I still improved on my 2014 forecasting, where for between 80%-20%, I was 47% accurate. This bears out something I highlighted last year: it’s really easy to predict the high and low risk countries, but it’s the ones in the middle that are difficult. MAIN TAKEAWAY: I’m getting better at forecasting, and while my level of forecasting does have some value, it still lacks the sort of predictive ability that I would like or would be obviously useful for policymakers.

Another thing I looked at last year was whether I was too optimistic or pessimistic about whether atrocities would occur. I’m interested in this because of the forecasting bias that makes people more likely to over-predict the likelihood of rare events and under-predict the likelihood of frequent events. Mass atrocities, of course, are extremely rare events. To do this, I’ll see about how many atrocities should have happened by adding up the percentage points I predicted. For example, in two cases, if I predicted a 95% likelihood in one and 5% likelihood in another, then out of the two, I predict one will happened (now I understand statistically this probably isn’t technically correct, but it’s close enough).

MAIN TAKEAWAY: If I do all that, I come up with a predicted atrocity total of 8.25, while there were 6 actual atrocities. So I over-predicted the likelihood of atrocities worldwide, but not terribly. I considerably under-predicted in 2014.

Before I conclude a few notes on several countries and measuring techniques. First, and perhaps most importantly, is how I determined whether a mass atrocity occurred. For many, like Syria or Zimbabwe, it was a no-brainer. For the ones I had any doubt about, I scoured the internet for figures, made some judgment calls when the figures weren’t clear or comprehensive, and then used my knowledge of the situation to determine whether the deaths were intentional. In short, my judgments are far from perfect, but so is the data. For some cases, there were UN or other reports with credible casualty figures, but I largely relied on ACLED. The problem with ACLED is it counts deaths conservatively, which means, for example, it lists the Janjaweed as only having killed about 4,000 civilians over the last 13 years in Darfur. One of the question marks I had, especially concerning the ACLED data was over Cameroon. ACLED listed 345 Cameroonian civilians killed by Boko Haram in 2015. That number seemed too low, especially considering there were reports more than 500 died in Fotokol alone. However, data from the Heidelberg Institute for International Conflict Research put the number at 422, while the Center for Complex Operations’ Hilary Matfess said that 1,200 people total had been killed in the conflict, but that included combatants. With that information at hand, I made the call to mark it a no.

For the second year running, I’ve been unable to determine whether a mass killing occurred in North Korea and Mexico. The Early Warning Project determines an ongoing mass killing perpetrated by North Korea against political opponents, but the information wall means I have no way to determine whether 1,000 died this year. As for Mexico, it’s too hard to know who counts as a civilian and whether any single cartel killed over 1,000.

Clearly, my biggest failing for this list was Yemen. I didn’t foresee the rapid advance of the Houthis and the resulting Saudi intervention that has resulted in thousands of deaths. Definitely a country to watch in 2016. On the other side of the spectrum, Sudan looks like a failure on my part because no atrocity occurred despite a 65% predicted probability. However I should note it’s quite hard to figure out exactly how many civilians government forces killed across Sudan. I couldn’t find enough evidence to determine with enough confidence that more than 1,000 people died across Sudan, but regardless, there was significant violence. In Pakistan, I also considerably overestimated the ability of jihadist groups to launch attacks in the wake of the Peshawar school massacre.

My forecasts for next year will go up Monday (assuming DC has not experienced the snowpocalypse).

One Response to “How’d I Do on My 2015 Mass Atrocity Forecasts?”


  1. 2016 Mass Atrocity Forecasts | The Widening Lens - January 25, 2016

    […] Friday’s post, I evaluated my predictions for 2015. In sum, I improved a bit on 2014, but still had some […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: