A year ago, I took a stab at predicting which countries would experience mass atrocities in 2014 (defined as 1,000 noncombatant intentional deaths caused by discrete group). My predictions were fairly accurate, if not perfectly so. Here’s what I predicted. I’ve put “YES” next to places that did experience atrocities and “NO” next to the countries that didn’t. For the countries where it’s simply too hard to know, I’ve put a “?”. I don’t want to get too in-depth into how I determined whether atrocities occurred, but I have some explanations in the footnotes for countries that are hard to judge one way or the other.
- Syria (95%) – YES
- South Sudan (85%) – YES
- Iraq (85%) – YES
- CAR (75%) – YES [1]
- Sudan (60%) – YES
- Afghanistan (50%) – YES [2]
- North Korea (50%) – ? [3]
- Mexico (35%) – ? [4]
- Nigeria (30%) – YES
- Burma (20%) – NO
- DRC (20%) – NO [5]
- Egypt (10%) – NO
- Mali (5%) – NO
- Venezuela (5%) – NO
To judge how accurate I was, one measure is to see each case as containing 100 points. If an atrocity did happen, then I get the number of percentage points that I predicted (for example, I get 95 out of 100 for Syria) and if one did not happen, I get the result of subtracting the number of percentage points I predicted from 100 (for example, I get 80 out of 100 for Burma). Because my predictions were not just yes/no, this method helps account for the probabilistic aspect. Measuring this way, I did very well, receiving 920 out of a possible 1200, excluding Mexico and North Korea because of the inconclusive judgments. However, that score should really be 920 out of 1400, because civilian deaths in Gaza during the Israel-Hamas conflict constitute a mass atrocity. Similarly, the Pakistani Taliban committed a mass atrocity. A mass atrocity may have occurred in Somalia, but the numbers don’t seem high enough to definitively say for sure.
There are a few problems with this metric for success, though. First, my numbers success rate is considerably boosted by the very high probability (the “No Shit List”) and the very low probability cases. If I remove the cases where I predicted probabilities above 80% and below 20%, and add in Pakistan and Gaza, my score comes out to a much less impressive 375 out of 800, even though by the standards of forecasting international events, it’s not bad.
The results of my projections have both optimistic and pessimistic ramifications for the ability to forecast atrocities. On the one hand, being a little less than 50% accurate in medium-risk cases is much better than the standard 65%-80% false positive ratio that’s common even in the best performing models (though it’s easier to outperform statistical models in one year than five). Additionally, with the exceptions of Pakistan and Gaza, no episodes of atrocities occurred in countries with probabilities less than 30%. On the other hand, in every case that I listed a probability that an atrocity would happen and it did, the country had been experiencing large-scale violent conflict at the beginning of 2014. One of the two cases I missed was also the one not experiencing large-scale violent conflict then.
Therein lies the problems. It’s fairly easy to predict where atrocities will occur for countries already experiencing mass violence. While it is certainly useful to predict anywhere where atrocities will occur, the real prize of forecasting is to identify the cases where atrocities will occur that aren’t obvious to the casual observer. Because mass atrocities are such rare events, that’s frustratingly difficult.
In my next post, I’ll put up my mass atrocity forecasts for 2015.
Update (1/16/15): Earlier today I realized that in analyzing my predictions I had missed the chance to analyze whether I had been overly optimistic or pessimistic about mass atrocities in 2014. I’m particularly interested to see if I avoided the bias that generally has forecasts over-predict the likelihood of rare events, which atrocities are.
I’ll do this by adding up the percentage points I predicted in total (and divide my 100) and then compare that to the actual occurrence of atrocities. If we exclude the atrocities that happened that I didn’t predict, I predicted there would be 5.4 mass atrocities in 2014. Within my prediction sample, there were actually 6 mass atrocities. So I was pretty close. My accuracy here was helped because each country that had a mass atrocity in 2014 in my predicted list also had one in 2013.
However, if I include Gaza and Pakistan (as I probably should), I was less accurate, again predicting 5.4 atrocities when 8 actually occurred. For whatever reason, I bucked the trend and under-predicted the number of atrocities that would occur in 2014.
Clarification (1/4/15): For this post, I defined a mass atrocity as 1,000 deaths in a single year. While this is partially consistent with other definitions for a mass atrocity used by The Early Warning Project and my thesis, it doesn’t clarify the conditions for when a mass atrocity continues over multiple years. The convention is that 1,000 is required in the onset year, and then if the number of deaths drops below a much lower threshold for a few years, then the mass killing episode ends. For example, by the Early Warning Project’s definition, a state-led mass killing episode continued in Myanmar last year, even though as far as I can tell, the casualty numbers were well under 1,000. For my predictions, because I’m only looking at one year at a time, I’m thinking about whether death counts will reach 1,000 each year. Neither definition is better than the other, but for the purposes of my predictions, the 1,000 threshold every year makes more sense.
[1] Though the numbers aren’t entirely clear, it seems very likely that more 1,000 noncombatants were killed by anti-balaka forces (and possibly ex-Seleka forces too) in 2014.
[2] By July, more than 1,500 civilians had already been killed, with 74% of those caused by anti-government forces (mostly the Taliban). The total number had risen to over 3,000 by November, with the Taliban responsible for 75%.
[3] Obviously, the North Korean regime isn’t releasing data on its prison camps, but investigations by Amnesty and the OHCHR makes it seem very likely more than 1,000 civilians died in 2014. However, the lack of data makes it impossible to know for sure.
[4] Like North Korea, there’s just not enough data to say. It’s not that we don’t know that huge numbers of people were killed by organized crime, but it’s unclear how many of those count as civilians (cartel members are combatants in this case). It seems likely, but one can’t be sure.
[5] While the civilian death toll almost certainly exceeded 1,000 in 2014, to my knowledge, no one single group can claim to have killed more than 1,000 noncombatants.
The Widening Lens 
Nice work, both on the forecasts and the public assessment of them. As you rightly note, predicting the changes (or transitions or phase shifts or whatever) is both the hardest and most important part.
FWIW, the most conventional measure for forecast accuracy is the Brier score, which you’ve kind of approximated here. There’s more than one version, but the simplest is the average of the differences between the observed outcome (scored as 1) and the probability assigned to it (on a 0-1 scale).