2015 Mass Atrocity Forecasts

In my last post, I looked back on how my predictions fared in 2014. While there are a couple different ways to measure success, all in all I was a little under 50%.

Here are my predictions for 2015. Like last time, I’ll not do a simple yes/no, but rather a percentage of how likely a mass atrocity is to happen. By mass atrocity, I mean 1,000 deaths caused intentionally by a discrete combatant group against another discrete noncombatant group. I’m including more countries than I did last year, and hopefully this will offer more accurate forecasts.

• Nigeria (95%)
• Iraq (95%)
• Syria (95%)
• Pakistan (75%)
• Afghanistan (70%)
• South Sudan (70%)
• Sudan (65%)
• Mexico (55%)
• CAR (50%)
• North Korea (50%)
• Somalia (30%)
• DRC (30%)
• Libya (25%)
• Gaza (25%)
• Cameroon (20%)
• Ukraine (10%)
• Rwanda (10%)
• Lebanon (10%)
• Burundi (5%)
• Yemen (5%)
• Chad (5%)
• Guinea (5%)
• Kenya (5%)
• Ethiopia (5%)
• Burma (5%)
• Eritrea (5%)
• Zimbabwe (5%)
• Mali (5%)

Explaining my forecast for each of the 28 countries here would be tedious and probably unnecessary, so I’ll skip it. However, I’ll select a few countries where my risk prediction doesn’t generally line up with the consensus in the atrocity prevention community.

In Nigeria, Boko Haram has probably already committed a mass killing in 2015, and across the border in Cameroon, Boko Haram is also active, though the chances of a mass atrocity are lower if not insignificant.

Pakistan and Afghanistan are not countries on the traditional atrocity prevention agenda, but that has more to do with uneasy relationship between anti-atrocity advocates and the U.S. military than the countries’ risk. Their respective Taliban’s both committed atrocities last year, and it seems likely that trend will continue.

In Mexico, it’s never a question of absolute casualty figures but how those casualties are categorized. Because there aren’t clear numbers on how many cartel members die as opposed to civilians, it’s hard to know whether more than 1,000 are killed by a specific drug cartel, even if thousands will almost certainly die in 2015.

In the DRC, like Mexico, more than 1,000 are highly likely to die. However, the splintered nature of armed groups in the country’s east means I think it’s more likely than not no single group will kill 1,000 civilians. The situation’s not dissimilar in Libya, where there is rampant violence, but it is committed by a myriad of militias.

Israel probably committed a mass killing in Gaza last year, and while confrontations between Hamas and Israel seem to operate on two or three year cycles, there’s still a decent chance Israel ‘mows the grass’ again this year.

While Rwanda is often praised as one of Africa’s most efficient governments, this sheen of good governance masks a political powder-keg. Whenever the elite coalition Kagame has built fractures, the struggle to fill the resulting power vacuum will likely result in mass violence. The same principle applies in Zimbabwe and Eritrea.

Finally, Burundi and Burma are two countries that have been high on the atrocity prevention agenda that I rated at only 5%. In Burundi, it seems the government has repressed the opposition enough that ruling elites are unlikely to be threatened during the 2015 election. There are some parallels here with Burma. While the treatment of the Rohingya minority is horrendous, it seems Burma’s elites have settled on forcing emigration rather than initiating a mass killing, which would be more politically risky.

Looking Back on My 2014 Mass Atrocity Forecasts

A year ago, I took a stab at predicting which countries would experience mass atrocities in 2014 (defined as 1,000 noncombatant intentional deaths caused by discrete group). My predictions were fairly accurate, if not perfectly so. Here’s what I predicted. I’ve put “YES” next to places that did experience atrocities and “NO” next to the countries that didn’t. For the countries where it’s simply too hard to know, I’ve put a “?”. I don’t want to get too in-depth into how I determined whether atrocities occurred, but I have some explanations in the footnotes for countries that are hard to judge one way or the other.

• Syria (95%) – YES
• South Sudan (85%) – YES
• Iraq (85%) – YES
• CAR (75%) – YES [1]
• Sudan (60%) – YES
• Afghanistan (50%) – YES [2]
• North Korea (50%) – ? [3]
• Mexico (35%) – ? [4]
• Nigeria (30%) – YES
• Burma (20%) – NO
• DRC (20%) – NO [5]
• Egypt (10%) – NO
• Mali (5%) – NO
• Venezuela (5%) – NO

To judge how accurate I was, one measure is to see each case as containing 100 points. If an atrocity did happen, then I get the number of percentage points that I predicted (for example, I get 95 out of 100 for Syria) and if one did not happen, I get the result of subtracting the number of percentage points I predicted from 100 (for example, I get 80 out of 100 for Burma). Because my predictions were not just yes/no, this method helps account for the probabilistic aspect. Measuring this way, I did very well, receiving 920 out of a possible 1200, excluding Mexico and North Korea because of the inconclusive judgments. However, that score should really be 920 out of 1400, because civilian deaths in Gaza during the Israel-Hamas conflict constitute a mass atrocity. Similarly, the Pakistani Taliban committed a mass atrocity. A mass atrocity may have occurred in Somalia, but the numbers don’t seem high enough to definitively say for sure.

There are a few problems with this metric for success, though. First, my numbers success rate is considerably boosted by the very high probability (the “No Shit List”) and the very low probability cases. If I remove the cases where I predicted probabilities above 80% and below 20%, and add in Pakistan and Gaza, my score comes out to a much less impressive 375 out of 800, even though by the standards of forecasting international events, it’s not bad.

The results of my projections have both optimistic and pessimistic ramifications for the ability to forecast atrocities. On the one hand, being a little less than 50% accurate in medium-risk cases is much better than the standard 65%-80% false positive ratio that’s common even in the best performing models (though it’s easier to outperform statistical models in one year than five). Additionally, with the exceptions of Pakistan and Gaza, no episodes of atrocities occurred in countries with probabilities less than 30%. On the other hand, in every case that I listed a probability that an atrocity would happen and it did, the country had been experiencing large-scale violent conflict at the beginning of 2014. One of the two cases I missed was also the one not experiencing large-scale violent conflict then.

Therein lies the problems. It’s fairly easy to predict where atrocities will occur for countries already experiencing mass violence. While it is certainly useful to predict anywhere where atrocities will occur, the real prize of forecasting is to identify the cases where atrocities will occur that aren’t obvious to the casual observer. Because mass atrocities are such rare events, that’s frustratingly difficult.

In my next post, I’ll put up my mass atrocity forecasts for 2015.

Update (1/16/15): Earlier today I realized that in analyzing my predictions I had missed the chance to analyze whether I had been overly optimistic or pessimistic about mass atrocities in 2014. I’m particularly interested to see if I avoided the bias that generally has forecasts over-predict the likelihood of rare events, which atrocities are.

I’ll do this by adding up the percentage points I predicted in total (and divide my 100) and then compare that to the actual occurrence of atrocities. If we exclude the atrocities that happened that I didn’t predict, I predicted there would be 5.4 mass atrocities in 2014. Within my prediction sample, there were actually 6 mass atrocities. So I was pretty close. My accuracy here was helped because each country that had a mass atrocity in 2014 in my predicted list also had one in 2013.

However, if I include Gaza and Pakistan (as I probably should), I was less accurate, again predicting 5.4 atrocities when 8 actually occurred. For whatever reason, I bucked the trend and under-predicted the number of atrocities that would occur in 2014.

Clarification (1/4/15): For this post, I defined a mass atrocity as 1,000 deaths in a single year. While this is partially consistent with other definitions for a mass atrocity used by The Early Warning Project and my thesis, it doesn’t clarify the conditions for when a mass atrocity continues over multiple years. The convention is that 1,000 is required in the onset year, and then if the number of deaths drops below a much lower threshold for a few years, then the mass killing episode ends. For example, by the Early Warning Project’s definition, a state-led mass killing episode continued in Myanmar last year, even though as far as I can tell, the casualty numbers were well under 1,000. For my predictions, because I’m only looking at one year at a time, I’m thinking about whether death counts will reach 1,000 each year. Neither definition is better than the other, but for the purposes of my predictions, the 1,000 threshold every year makes more sense.

                                                                                                                                                                                                                     

[1] Though the numbers aren’t entirely clear, it seems very likely that more 1,000 noncombatants were killed by anti-balaka forces (and possibly ex-Seleka forces too) in 2014.

[2] By July, more than 1,500 civilians had already been killed, with 74% of those caused by anti-government forces (mostly the Taliban). The total number had risen to over 3,000 by November, with the Taliban responsible for 75%.

[3] Obviously, the North Korean regime isn’t releasing data on its prison camps, but investigations by Amnesty and the OHCHR makes it seem very likely more than 1,000 civilians died in 2014. However, the lack of data makes it impossible to know for sure.

[4] Like North Korea, there’s just not enough data to say. It’s not that we don’t know that huge numbers of people were killed by organized crime, but it’s unclear how many of those count as civilians (cartel members are combatants in this case). It seems likely, but one can’t be sure.

[5] While the civilian death toll almost certainly exceeded 1,000 in 2014, to my knowledge, no one single group can claim to have killed more than 1,000 noncombatants.

I’m Not That Great a Forecaster: Looking back on my past predictions and learning how to improve

In early January of 2013, I wrote two posts that outlined six conflicts to watch for civilian protection advocates in the coming year.  Without any concrete methodology, I picked out Sudan, Afghanistan, Mali, Kenya, Syria (specifically violence in a post-Assad Syria), and Central Asia.  Arguably, I was wrong in five of the six cases if the criteria is that the violence had to get significantly worse from 2012 to 2013 in the manner that I predicted  (it gets even worse when you think about all 2013 conflicts I omitted; Nigeria, Egypt, CAR, and Iraq all experienced episodes of mass killing that have intensified since 2012).  I’ll briefly outline how I did country by country, address what I did wrong, and because it’s that time of year again, propose predictions for 2014.  If you don’t want to read the whole thing, feel free to skip to the bulleted predictions.

Sudan had a turbulent year, but it’s nothing really out of the ordinary for the troubled country.  #SudanRevolts returned in September and October and prompted a fairly superficial cabinet reshuffle, but not much else.  Violence continued to rage in Darfur, South Kordofan, and Blue Nile.  Although violence increased in Darfur and perhaps South Kordofan, it was not a radical departure for 2012.  In my last sentence of my Sudan section, I briefly mentioned South Sudan.  While the violence in Jonglei between the Lou Nuer and Murle ebbed slightly in 2013, any progress made in the country was obliterated by the crisis that began on December 15th and has enveloped the country.  It’s unclear how many people have died, but it probably constitutes a mass killing. Mali has certainly experienced some violence in 2013, but there is no way that it was worse than 2012.  The French-African intervention was fairly successful at expelling the jihadist-Tuareg alliance from Northern Mali.  Fairly successful elections went ahead and the leader of the initial coup, Amadou Haya Sanogo, has been arrested and will be put on trial.

Afghanistan is probably the only case I got right.  Civilian casualties increased in the first half (and possibly the second) of 2013, marking a reversal in 2012’s trend.  For every success there’s a failure, and Kenya is that failure.  The March elections went off smoothly, and so I was really wrong.  I was right to predict that 2013 would be worse than 2012 for Syrians, but it didn’t happen in the way I thought.  At the time, it seemed very likely Assad would fall, initiating a mass killing of Alawites in and around Latakia.  That didn’t happen.  Instead, massive violence continued against civilian populations throughout Syria mostly with the exception of Latakia.  Finally, like Kenya, I really got Central Asia wrong.  There was not significant violence in any Central Asian country (excluding Afghanistan) this calendar year.  Regime change could have potentially caused conflict, but those pesky autocratic regimes just won’t go away.

So how can I improve?  First, it would have helped if I had had a concrete measurement for what constituted conflict.  Second, it would have made sense to have had a constant metric for assessing why I thought conflict would occur in certain places.  Figuring out what my predicted conflict zones had in common and why they were different from other potential conflict zones would have improved my methodology, even if creating a model from a hunch isn’t the best social science (if someone wants to pay me to blog I promise my methodology will be more robust).  My analysis also would have improved if I had laid out specifically what changes I was looking for and how they fit into a larger historical narrative.  For example, while there was both a history of and a potential for political instability in Central Asia, my only data points were the 2005 massacre in Andijan, Uzbekistan and the 2010 violence in Osh, Kyrgyzstan.

So moving into my predictions for 2014, rather than creating a complete methodology of my own, I’m going to borrow two of Jay Ulfelder’s crowd-sourced forecasting projects as points of reference.  The first is the Center for Genocide Prevention’s online opinion pool (password needed).  The opinion pool aggregates the opinions of currently fifty individuals interested in atrocity prevention to create averaged probabilities about the likelihood of a particular event.  The questions are generally phrased like this, “Before X date, will an episode of state-led mass killing occur in Y country.”  The second is a wiki survey also originating from the Center for Genocide Prevention.  The results demonstrate how much more likely any particular country is to experience an episode of state-led mass killing than other countries during 2014.

In order and with percentages, these are the countries that I think are most likely to experience a mass killing (defined as more than 1,000 civilian deaths) in 2014:

• Syria (95%)
• South Sudan (85%)
• Iraq (85%)
• CAR (75%)
• Sudan (60%)
• Afghanistan (50%)
• North Korea (50%)
• Mexico (35%)
• Nigeria (30%)
• Burma (20%)
• DRC (20%)
• Egypt (10%)
• Mali (5%)
• Venezuela (5%)

My predictions are largely based on the crowd-sourced forecasts from Jay’s two projects, aren’t identical.  I’ll explain why, country by country, below.

Obviously, the chances that a mass killing will happen in Syria are very, very high (the wiki survey is definitely wrong in placing Syria 6th).  However, the opinion pool’s average probability that there will be a mass killing of Alawite civilians in Latakia province stands at 43%.  I think this is too high, and the real probability stands nearer 15%.  For a mass killing of Alawites to happen, the Assad regime would have to collapse or severely weaken.  Demonstrated by my false prediction of his doom in early 2013, Assad has proven surprisingly durable.  Civil wars tend to last a long time, so especially with the increasing fragmentation of the opposition, it’s doubtful Assad will be defeated anytime soon.

South Sudan, Iraq, and CAR all have ongoing conflict that will almost certainly include a case of mass killing, committed either by government or rebel forces, in 2014.  Iraq and CAR definitely experienced a mass killing episode in 2013, and South Sudan probably did, but the concrete numbers to confirm it don’t exist.  I pegged the chances of a future mass killing as slightly lower in CAR only because of the combination of the peacekeeping force and the higher potential for resolution than in South Sudan.

Jay Ulfelder, in his review of mass killing in 2013, wrote of Sudan, “…where the uncertainty is not whether the regime is engaging in mass killing but in how many parts of the country at once and targeting how many different groups.”  He’s right, and unfortunately civilians in Darfur, South Kordofan, and Blue Nile will likely continue to suffer in 2014.  In the opinion pool, a question asks the probability that Sudan will carry out a mass killing of anti-government activists will occur in 2015, and places the average at 31%.  I think this is far, far too high.  Despite significant anti-government protests, the body count has always remained low.  The Sudanese regime is intelligent in this respect, instead opting for mass arrests of protest leaders.  The scarcity of bloody street confrontations decreases the potential of igniting the paradox of repression.  It is also telling that the bloodiest anti-government protest this year happened in Nyala, South Darfur.  Khartoum is happy to take the fight to enemies in the periphery, but remains wary of the threat from the center.  If the government were to significantly weaken, there’s an increased chance it would unleash lethal violence against anti-government activists.  However, despite increasing organization from the political opposition and calls that the regime was about to fall, the NCP remains in power.

Afghanistan probably experienced a mass killing with the Tailban as the perpetrators in 2013, and there aren’t a lot of positive signs for the future.  In North Korea, it’s basically impossible to tell how many people are dying each year in giant concentration camps.  50% is simply a wild guess.

Drug violence in Mexico is out of control, but I’m hesitant to peg a high probability of a mass killing because it’s unclear what share of civilians vs. cartel members are killed in drug violence (in a tweet, Jay confirmed that cartel members count as combatants).  Bottom line: a lot of people will die in Mexico in 2014, but it may or may not constitute a mass killing.

Nigeria experienced a classic case of a counterinsurgent mass atrocity last year, and despite some international attention on the issue, there is still a decent chance it happens again.  Nigeria is 14th on the wiki survey for the chances of state-led killing (too low in my opinion) for 2014 but doesn’t appear as a question in the opinion pool.

Burma is a country that is very much in the news for people interested in atrocity prevention, but I’m more optimistic than other analysts about the prospects for 2014.  Burma’s counterinsurgencies against ethnic-minority armies are long-running, but have recently had quite low body counts.  I see no reason for that trend to stop.  The opinion pool predicts there is a 35% chance of a mass killing of Rohingya in 2014.  No single rioting incident has yet escalated to widespread killing (the Meiktila incident killed dozens, not hundreds).  Despite the massive persecution the Rohingya face, the levels of lethal violence have remained relatively low.  Without an obvious trigger, I think more slow-motion ethnic cleaning is far more likely than a full-blown mass killing in 2014.

DRC placed fourth in the wiki survey.  Perhaps this is a bit excessive, but not completely unwarranted.  The combination of a country in transition from autocracy to democracy, the prevalence of armed groups in the Kivus, and intrusive neighbors means the risk of a mass killing in the DRC remains relatively high.

Egypt also suffers from an unfortunate history.  Last year, the government undertook a mass killing in response to pro-Morsi demonstrations.  A similar scenario could repeat itself, violence in the Sinai could rapidly increase, or even less likely, a civil war that involves mass killing could erupt.  All of this is plausible, if not incredibly likely (Egypt is 15th in the wiki survey).

The situation is not absolutely analogous to the DRC’s, but Mali also suffers from a somewhat unstable post-major conflict environment.  The opinion pool average for a state-led mass killing rests at 13%.  I think this is too high (probably suffering from the bias that comes from forecasting rare events) because of the progress Mali has made since 2012, but not terribly so.  In the wiki survey, Mali is 3rd, which strikes me as overly pessimistic.

Finally, Venezuela is a bit of a stretch, but I decided to include it anyways.  Few atrocity prevention advocates are paying attention despite the high political instability and absolutist rhetoric coming out of the Maduro administration.  Though Venezuela appears 82nd on the wiki survey and isn’t in a region of the world that has been prone to mass killing recently, I think a political crisis resulting in a government mass killing is plausible if still very unlikely.

Correction: Jay Ulfelder wrote this in a comment, “One point of clarification about Syria and some of the other cases you discuss: in both the statistical modeling and the wiki survey, we’re looking at the risk that a *new episode* of mass killing will start, not the risk that the one(s) we’re seeing now will continue. So Syria could hypothetically get a very low predicted probability or rank if the models or crowd deemed it unlikely that the state would begin deliberately killing large numbers of civilians from a discrete group it isn’t already targeting now. Hence the question in the Syrian case about Alawites but not one about the groups the regime is killing in large numbers now.”  I didn’t realize that when I wrote the post.

Musings on the Excessive Silliness of Red Dawn

It’s a senior week at Swarthmore, so I have a lot of free time.  With that free time, I watched the 2012 remake of the 1984 classic movie Red Dawn.  In the Cold War version, the Soviets and Cubans invade, but are partially fended off by a group of Colorado high-schoolers.  In the new (and probably not improved) film, it’s the sinister North Koreans invading, and the setting is shifted to Spokane, Washington.  The movie includes a plethora of implausible scenes, but also sparked some good discussion among the four of us that watched the movie.  I can’t take credit for all the below ideas, so this post was more of a collaborative effort (thanks Taryn, Lorand, and Sam) that I’d like to share.

Unlike in 1984, it’s pretty hard to identify an obvious invasion threat to the United States, and therefore the filmmakers were forced to create a three minute montage in the beginning to set the stage for a North Korean invasion.  Everything from the bailout in Greece, Islamic terrorist in Russia, cyberwarfare, and new conflicts in the Middle East were thrown into the mix to create a sense of impending crisis.  While it was a hoot to watch, it probably didn’t really help the movie at the box office.  The film’s target audience, middle America, is unlikely to fully understand the complex international problems that were subsequently exaggerated in order to make the movie somewhat plausible.  While NK is does exist in the American public consciousness as a threat, it isn’t comparable to the imagined threat previously posed by the USSR.  The filmmakers were forced to cast North Korea as the bad guys to give the movie a chance at the box office in China, but even China, if potentially more scary, doesn’t carry the same ideological threat that the Soviets did.  The threat of communism was enough to get people on the edge of their seats, but there isn’t a single word that can describe the Chinese threat to the American way of life in the same way.

In the film, following the North Korean invasion, our heroes (who adopt the name “Wolverines”) escape their cookie-cutter suburb and make for the wooded hills, where they set up a base from which to launch attacks on the occupiers.  While they can be forgiven for being disorganized both organizationally and strategically in the beginning of the film, their strategy fails to develop.  Their only goal seems to “resist” by killing North Korean soldiers.  The group’s rhetoric is peppered by references to freedom and patriotism, rather than a fixed final goal they are working toward.  Though they attempt to contact American central command through a North Korean EMP-resistant communication device, the group failed to use the radios they had all along.  They never make any effort to link up with other guerrilla groups, and only begrudgingly accept help from Marines they run into.  Similarly, while they seek help from people in town when necessary, they never attempt to recruit new members until the final scenes of the film.  Finally, in one of the most interesting scenes, the group’s leader Jed, a former Marine who had previously fought in Iraq, gives the group a quick rundown of guerrilla warfare.  He says that in Iraq, he was one of the good guys who helped to enforce law and order.  Now however, Jed says, the Wolverines, are the bad guys, who need to unsettle their North Korean occupiers.  Despite using moral rhetoric, the speech actually totally avoids the tricky issues of morality at play as the audience is supposed to accept that no matter where they are in the world or what they are doing, Americans are the good guys.

While it’s not the most glaring, the film’s portrayal of NK’s counterinsurgency (COIN) tactics is its most absurd aspect.  Firstly, North Korean paratroopers parachute in the Wolverines’ tranquil suburb, but miraculously avoid trees and power lines.  Once firmly in power, the first action is to set up reeducation camps (the cliche variety complete with loudspeakers extolling the errors of American society) for a randomly selected group of people, which prove to be entirely ineffective.  Over the course of the film, NK’s COIN strategy is ultimately very gentle.  There are only two portrayed executions of American resisters (another is implied), and excluding numerous checkpoints, there seems not to be any punishment handed down to American residents despite the Wolverines’ repeated attacks.  This strange facts comes down to the American public not understanding how COIN works.  Collective punishment is never even threatened in the film, and while North Koreans are quite gruff in their dealings with Americans, this film is not Battle of Algiers.  Finally, NK is entirely incompetent as an occupier.   Though less than ten people make up the Wolverines, North Korea is unable to stop them from moving in and out of town despite knowing their identities.  Finally, the Wolverines are able to inflict very disproportionate losses on North Korean troops despite have inferior technology, almost no formal training, and a much smaller fighting force.

Red Dawn’s portrayal of military engagement is the most obvious object of criticism, especially for people watching it with an interest in political violence, but the movie also has a race problem.  There are three black characters: Mayor Jenkins, his son Daryl, and Wolverines member Danny.  They fulfill three common stereotypical roles for black actors: a morally weak Uncle Tom, the first-to-die, and the selfless yet underdeveloped individual who sacrifices himself for the (white) group.  Mayor Jenkins is one of the first Americans to defect to the North Korean side, and meekly attempts to convince the Wolverines to surrender.  Danny is the first Wolverine to die, and we learn little about him before his death.  Daryl survives until the end of the film, but inexplicably decides to stay behind and fight on alone after being implanted with a tracking device that endangers the group.  There is no attempt made to remove the device before the decision to leave him behind is made.

The film is not a cinematic masterpiece, but is worth a viewing solely for the number of laughs it produces.