I’m Not That Great a Forecaster: Looking back on my past predictions and learning how to improve

In early January of 2013, I wrote two posts that outlined six conflicts to watch for civilian protection advocates in the coming year.  Without any concrete methodology, I picked out Sudan, Afghanistan, Mali, Kenya, Syria (specifically violence in a post-Assad Syria), and Central Asia.  Arguably, I was wrong in five of the six cases if the criteria is that the violence had to get significantly worse from 2012 to 2013 in the manner that I predicted  (it gets even worse when you think about all 2013 conflicts I omitted; Nigeria, Egypt, CAR, and Iraq all experienced episodes of mass killing that have intensified since 2012).  I’ll briefly outline how I did country by country, address what I did wrong, and because it’s that time of year again, propose predictions for 2014.  If you don’t want to read the whole thing, feel free to skip to the bulleted predictions.

Sudan had a turbulent year, but it’s nothing really out of the ordinary for the troubled country.  #SudanRevolts returned in September and October and prompted a fairly superficial cabinet reshuffle, but not much else.  Violence continued to rage in Darfur, South Kordofan, and Blue Nile.  Although violence increased in Darfur and perhaps South Kordofan, it was not a radical departure for 2012.  In my last sentence of my Sudan section, I briefly mentioned South Sudan.  While the violence in Jonglei between the Lou Nuer and Murle ebbed slightly in 2013, any progress made in the country was obliterated by the crisis that began on December 15th and has enveloped the country.  It’s unclear how many people have died, but it probably constitutes a mass killing. Mali has certainly experienced some violence in 2013, but there is no way that it was worse than 2012.  The French-African intervention was fairly successful at expelling the jihadist-Tuareg alliance from Northern Mali.  Fairly successful elections went ahead and the leader of the initial coup, Amadou Haya Sanogo, has been arrested and will be put on trial.

Afghanistan is probably the only case I got right.  Civilian casualties increased in the first half (and possibly the second) of 2013, marking a reversal in 2012’s trend.  For every success there’s a failure, and Kenya is that failure.  The March elections went off smoothly, and so I was really wrong.  I was right to predict that 2013 would be worse than 2012 for Syrians, but it didn’t happen in the way I thought.  At the time, it seemed very likely Assad would fall, initiating a mass killing of Alawites in and around Latakia.  That didn’t happen.  Instead, massive violence continued against civilian populations throughout Syria mostly with the exception of Latakia.  Finally, like Kenya, I really got Central Asia wrong.  There was not significant violence in any Central Asian country (excluding Afghanistan) this calendar year.  Regime change could have potentially caused conflict, but those pesky autocratic regimes just won’t go away.

So how can I improve?  First, it would have helped if I had had a concrete measurement for what constituted conflict.  Second, it would have made sense to have had a constant metric for assessing why I thought conflict would occur in certain places.  Figuring out what my predicted conflict zones had in common and why they were different from other potential conflict zones would have improved my methodology, even if creating a model from a hunch isn’t the best social science (if someone wants to pay me to blog I promise my methodology will be more robust).  My analysis also would have improved if I had laid out specifically what changes I was looking for and how they fit into a larger historical narrative.  For example, while there was both a history of and a potential for political instability in Central Asia, my only data points were the 2005 massacre in Andijan, Uzbekistan and the 2010 violence in Osh, Kyrgyzstan.

So moving into my predictions for 2014, rather than creating a complete methodology of my own, I’m going to borrow two of Jay Ulfelder’s crowd-sourced forecasting projects as points of reference.  The first is the Center for Genocide Prevention’s online opinion pool (password needed).  The opinion pool aggregates the opinions of currently fifty individuals interested in atrocity prevention to create averaged probabilities about the likelihood of a particular event.  The questions are generally phrased like this, “Before X date, will an episode of state-led mass killing occur in Y country.”  The second is a wiki survey also originating from the Center for Genocide Prevention.  The results demonstrate how much more likely any particular country is to experience an episode of state-led mass killing than other countries during 2014.

In order and with percentages, these are the countries that I think are most likely to experience a mass killing (defined as more than 1,000 civilian deaths) in 2014:

• Syria (95%)
• South Sudan (85%)
• Iraq (85%)
• CAR (75%)
• Sudan (60%)
• Afghanistan (50%)
• North Korea (50%)
• Mexico (35%)
• Nigeria (30%)
• Burma (20%)
• DRC (20%)
• Egypt (10%)
• Mali (5%)
• Venezuela (5%)

My predictions are largely based on the crowd-sourced forecasts from Jay’s two projects, aren’t identical.  I’ll explain why, country by country, below.

Obviously, the chances that a mass killing will happen in Syria are very, very high (the wiki survey is definitely wrong in placing Syria 6th).  However, the opinion pool’s average probability that there will be a mass killing of Alawite civilians in Latakia province stands at 43%.  I think this is too high, and the real probability stands nearer 15%.  For a mass killing of Alawites to happen, the Assad regime would have to collapse or severely weaken.  Demonstrated by my false prediction of his doom in early 2013, Assad has proven surprisingly durable.  Civil wars tend to last a long time, so especially with the increasing fragmentation of the opposition, it’s doubtful Assad will be defeated anytime soon.

South Sudan, Iraq, and CAR all have ongoing conflict that will almost certainly include a case of mass killing, committed either by government or rebel forces, in 2014.  Iraq and CAR definitely experienced a mass killing episode in 2013, and South Sudan probably did, but the concrete numbers to confirm it don’t exist.  I pegged the chances of a future mass killing as slightly lower in CAR only because of the combination of the peacekeeping force and the higher potential for resolution than in South Sudan.

Jay Ulfelder, in his review of mass killing in 2013, wrote of Sudan, “…where the uncertainty is not whether the regime is engaging in mass killing but in how many parts of the country at once and targeting how many different groups.”  He’s right, and unfortunately civilians in Darfur, South Kordofan, and Blue Nile will likely continue to suffer in 2014.  In the opinion pool, a question asks the probability that Sudan will carry out a mass killing of anti-government activists will occur in 2015, and places the average at 31%.  I think this is far, far too high.  Despite significant anti-government protests, the body count has always remained low.  The Sudanese regime is intelligent in this respect, instead opting for mass arrests of protest leaders.  The scarcity of bloody street confrontations decreases the potential of igniting the paradox of repression.  It is also telling that the bloodiest anti-government protest this year happened in Nyala, South Darfur.  Khartoum is happy to take the fight to enemies in the periphery, but remains wary of the threat from the center.  If the government were to significantly weaken, there’s an increased chance it would unleash lethal violence against anti-government activists.  However, despite increasing organization from the political opposition and calls that the regime was about to fall, the NCP remains in power.

Afghanistan probably experienced a mass killing with the Tailban as the perpetrators in 2013, and there aren’t a lot of positive signs for the future.  In North Korea, it’s basically impossible to tell how many people are dying each year in giant concentration camps.  50% is simply a wild guess.

Drug violence in Mexico is out of control, but I’m hesitant to peg a high probability of a mass killing because it’s unclear what share of civilians vs. cartel members are killed in drug violence (in a tweet, Jay confirmed that cartel members count as combatants).  Bottom line: a lot of people will die in Mexico in 2014, but it may or may not constitute a mass killing.

Nigeria experienced a classic case of a counterinsurgent mass atrocity last year, and despite some international attention on the issue, there is still a decent chance it happens again.  Nigeria is 14th on the wiki survey for the chances of state-led killing (too low in my opinion) for 2014 but doesn’t appear as a question in the opinion pool.

Burma is a country that is very much in the news for people interested in atrocity prevention, but I’m more optimistic than other analysts about the prospects for 2014.  Burma’s counterinsurgencies against ethnic-minority armies are long-running, but have recently had quite low body counts.  I see no reason for that trend to stop.  The opinion pool predicts there is a 35% chance of a mass killing of Rohingya in 2014.  No single rioting incident has yet escalated to widespread killing (the Meiktila incident killed dozens, not hundreds).  Despite the massive persecution the Rohingya face, the levels of lethal violence have remained relatively low.  Without an obvious trigger, I think more slow-motion ethnic cleaning is far more likely than a full-blown mass killing in 2014.

DRC placed fourth in the wiki survey.  Perhaps this is a bit excessive, but not completely unwarranted.  The combination of a country in transition from autocracy to democracy, the prevalence of armed groups in the Kivus, and intrusive neighbors means the risk of a mass killing in the DRC remains relatively high.

Egypt also suffers from an unfortunate history.  Last year, the government undertook a mass killing in response to pro-Morsi demonstrations.  A similar scenario could repeat itself, violence in the Sinai could rapidly increase, or even less likely, a civil war that involves mass killing could erupt.  All of this is plausible, if not incredibly likely (Egypt is 15th in the wiki survey).

The situation is not absolutely analogous to the DRC’s, but Mali also suffers from a somewhat unstable post-major conflict environment.  The opinion pool average for a state-led mass killing rests at 13%.  I think this is too high (probably suffering from the bias that comes from forecasting rare events) because of the progress Mali has made since 2012, but not terribly so.  In the wiki survey, Mali is 3rd, which strikes me as overly pessimistic.

Finally, Venezuela is a bit of a stretch, but I decided to include it anyways.  Few atrocity prevention advocates are paying attention despite the high political instability and absolutist rhetoric coming out of the Maduro administration.  Though Venezuela appears 82nd on the wiki survey and isn’t in a region of the world that has been prone to mass killing recently, I think a political crisis resulting in a government mass killing is plausible if still very unlikely.

Correction: Jay Ulfelder wrote this in a comment, “One point of clarification about Syria and some of the other cases you discuss: in both the statistical modeling and the wiki survey, we’re looking at the risk that a *new episode* of mass killing will start, not the risk that the one(s) we’re seeing now will continue. So Syria could hypothetically get a very low predicted probability or rank if the models or crowd deemed it unlikely that the state would begin deliberately killing large numbers of civilians from a discrete group it isn’t already targeting now. Hence the question in the Syrian case about Alawites but not one about the groups the regime is killing in large numbers now.”  I didn’t realize that when I wrote the post.

Advertisement

Adding Nuance to the Peace vs. Justice Debate

The peace versus justice debate is unavoidable when it comes to the International Criminal Court (ICC).  The conversation goes something like: Team Peace argues that the immediate cessation of violent conflict has to take precedence over everything else, while Team Justice argues that ending impunity for human rights violations is crucial for deterrence against human rights violations in the future.  While this summary totally simplifies a complicated and multipolar conversation, these two camps shape the basic nature of the debate.  Though both have solid points, a messy, subjective truth lies somewhere in the middle and the effects of justice are heavily dependent on the specific situation.

While the division between peace and justice is not rock-solid, there are indeed real problems with pursuing justice over peace (a theme I’ve written about before).  A perfect example is Sudan.  The ICC’s arrest warrant against President Omar al-Bashir increases his need to stay in a position of power (though he says he will step down in 2015 this is probably more of a result of internal NCP politics and he certainly has no intention of handing himself over to the ICC), and has decreased his ability to participate in negotiations.  This fact decreases the possible avenues of engagement for the international community (to varying degrees depending on the actor) to bargain with Bashir, ultimately hampering the opportunities for an international tempering influence, which his is especially unfortunate given Bashir’s current position of weakness.

Another example of unintended ICC consequences is in Kenya, where ICC-charged duo Uhuru Kenyatta and William Ruto were essentially brought together as a political unit because of their respective warrants that date back to the election violence in 2007-2008.  Ruto and Kenyatta were able to use their confrontation with the ICC as a symbol of their resistance against foreign influence, consequently gaining them votes.  Their ticket eventually won the Kenyan elections (though there seems to also be evidence that the ICC helped convince Kenyatta and Ruto to call for calm before and after the election), and Kenyatta is now the second head of state to have been summoned by the ICC.  Unlike Bashir however, Kenyatta has cooperated with The Hague thus far.

So while there are real downsides to justice over peace, there are also plenty of benefits from a justice-centered approach.  As Erik Voeten points out in his Monkey Cage post, the ICC is very effective in deterring human rights abuses in countries where “mid-level” human rights abuses take place.  Also, the ICC is quite good at influencing mid-level individuals.  While Bashir, as Sudan’s leader, is out of the ICC’s reach, mid-level individuals in security forces and rebel groups worldwide are much more expendable, and they know that if a higher-up decides they’re a liability because of the atrocities they’ve committed, they’ll be on the next plane to The Hague.  The threat of ICC prosecution, for example, had a positive effect in Colombia, and the institution is quite effective at deterring torture.

Unfortunately though, the existence of the ICC does little to deter the most egregious human rights violations.  Individuals like Assad, Bashir, and Gaddafi have never been cowed by threats of eventual justice.  Keeping power outweighs any potential risks.  Conversely though, the existence of the ICC does not encourage human rights violations as James Fearson argued.  While it is supposed to, the ICC does not really close off all escape routes (they’ve never been in short supply anyway) for human rights violators, but these escape routes, in the end, have little effect on the level of human rights violations committed.  It is not as if Assad is being encouraged to kill as many people as possible before escaping to the ICC; leaders like Assad and Gaddafi never had any intention of pulling the escape cord when it looked like they have a credible chance of losing.  While the ICC can do little to prevent leaders bent on maintaining power through any means necessary from doing so, it can at least provide a just conclusion to some of these cases (Bosco Ntaganda is a good example), an outcome which shouldn’t be trivialized.

Justice and peace are not mutually exclusive phenomena, and while one can sometimes endanger the other, the specific context of each situation must always be taken into account before making a policy recommendation.  Ultimately, this is a debate that the ICC will have to enter to an increasing degree in coming years.  While it has made some progress, it must to do more to address the problems that come with an inflexible, justice-centered approach.  Luckily, it does have the tools to do that.  Article 53 of the Rome Statute, the founding document of the ICC, allows for the Chief Prosecutor to offer amnesty to a perpetrator in the interests of the victims.  This precedent should not be applied in every situation, but does potentially allow the ICC to take a more critical approach to its activities.  The ICC has certainly been a milestone achievement in the fight to end international impunity for large-scale human rights violations, but it is not without its problems.

The Coup in Sudan: Some Theories

Though the entire incident is shrouded in confusion, it appears as there was attempted coup in Sudan on November 22nd.  Though the coup did not have any real impact on violence levels or the situation of Sudan’s civilians, the potential impact of regime change on  in these sectors is enormous, and therefore it is an important issue to keep tabs on for civilian protection advocates.  The details of the event are still mostly unclear, and so while it is impossible to establish a comprehensive summary, there are a few ideas that I would like to put forward.  Inevitably, with time, the events of November 21st and 22nd will become more clear.

One theory regarding the incident was that it was simply a government hoax aimed to repressing dissent, and there was never a real threat to Bashir’s regime.  This doesn’t make sense for a few reasons.  Firstly, those arrested come from within the NCP, despite attempts to obfuscate.  Khartoum has no reason to publicize splits within the NCP if it doesn’t have to.  Secondly, the messaging in the early hours following the coup was confused.  Numerous government officials made conflicting statements about who was involved, who was arrested, etc.  Presumably, an government sponsored hoax would have been preceded by inter-agency cooperation and planning, and this seems not to have happened.  While this fact seems to almost certainly point to a putsch attempt, it at least indicates the severe lack of institutional organization within the regime.

While the coup has not led to overthrow of Bashir, it has significant analytical importance for examining the state of the Bashir regime.  Currently, Sudan is experiencing severe economic problems, which culminated in #SudanRevolts this summer.  Additionally, there are doubts about the health of Bashir, who is recovering from a throat operation.  Finally, the NCP is experiencing severe political divisions, with roughly the hard-line Islamists on one side and the reformers on the other.  These three factors, combined with the attempted coup, reveal the extreme vulnerability afflicting the current Sudanese regime.  Theda Skocpol, and other structuralists, has highlighted the importance of state collapse in precipitating revolutions.  Therefore, not only can regimes not fall without debilitating internal strife, but revolutionary movements can only exist if there are signs of state weakness.  If we accept this line of theory, we can expand her analytical framework to encompass coups, and conclude there could never have been an attempted coup without significant state weakness.  The regime is currently not fragile enough to crumble under its own weight, but it is weak enough to produce an attempted coup.

Despite obvious signs of state weakness, Bashir’s security forces responded promptly to the attempted putsch.  Though it seems clear the plotters were hard-liners from Al-Sae’ohoon, the government clumsily and belatedly tried to blame the opposition.  While the coup response was intended to strengthen and consolidate the regime, it seems to have only further isolated Bashir and his cronies.  One major question, however, is how active and deliberate was this coup?  Bashir clearly felt the need to crackdown, but was this because of an imminent coup or simply deep divisions within the NCP?  It is quite plausible that Salah Gosh was indeed organizing opposition within the NCP, but did not intend to overthrow the government.  Did Gosh really put everything on the line as the government has claimed, or was he merely working toward shifting the party’s policies in a more hard-line direction?  With the current information, it’s basically impossible to tell.

There, of course, is a long history of failed coups all around the world.  However, I think it is important, especially in the Sudanese context, to not merely dismiss the coup as a failure, but to place it within a broader context.  The current state weakness may indicate that this was a “dress rehearsal” coup.  What I mean here is that this coup was a first attempt to overthrow the government, which gives the opposition a reading on how the regime would respond to a later coup.  This can be interpreted in two ways.  Firstly, the opposition simply wanted to test out the strength of the regime, and therefore put a small group of individuals together to challenge the regime, with the knowledge that it would likely fail.  This model, however, seems overly conspiratorial.  Secondly, the weakness of the regime encouraged a small group of plotters to try to overthrow the government, but were unable to garner enough support due to others’ readings of state strength.  Opposing factions now know how the government will react to a coup, and may be able to create a larger, successful coup in the future.

“Dress rehearsal” coups have happened before.  Argentina in 1955 and Chile in 1973 both provide examples of an attempt by a portion of the armed forces to overthrow the government that failed.  In both cases, the armed forces united three months later and succeeded.  Could this happen in Sudan?  While the situation is not totally analogous, there are certainly similarities.  Like Argentina and Chile, Sudan is in a state of crisis and the public has a growing lack of trust in the government.  The initial attempt to seize the state seems to have been caused by the apparent fragility of the government, though it was too soon to actually gather support.  Ultimately, the failed coup attempt weakened the regime, and the armed forces saw that they would indeed be able to seize power.  In Sudan, we have a failed coup that included parts of the armed forces at a time of great state weakness, causing a backlash from the ruling faction.  In response to this coup, Bashir has seemingly isolated himself from both Islamists and the opposition by imprisoning the first and blaming the second.  A successful coup in the future certainly isn’t inevitable, but Bashir shouldn’t count himself lucky just yet.

* I am not a political scientist, therefore any comments or corrections on the theories presented in the last few paragraphs would be appreciated.