ISAF has taken ‘serious issue’ with AAN’s latest report ‘A Knock on the Door. 22 Months of ISAF Press Releases’ (see full text of the press release below). ISAF argues that the methodology is flawed and based on incomplete information, and it worries that this will ‘confuse serious researchers or those engaged in balanced reporting on this subject.’ The response, and the argument ISAF has with the AAN report, illustrates some of the key issues surrounding the attempts at information management and strategic communications by ISAF and others.
The AAN report, by Kandahar-based researchers Alex Strick van Linschoten and Felix Kuehn, analyses all information surrounding deaths and detentions during military operations contained in the ISAF press releases from 1 December 2009 to 30 September 2011. The report then zooms in on the deaths and detentions during the so-called capture-or-kill operations and compares this information, in particular with regard to the number of ‘leaders’ and ‘facilitators’, with the total figures that are periodically released.
The main questions the report sought to answer, based on the available information, were these: How many raids are taking place? How many people are being detained and how many are being killed? How are operations distributed geographically and over time? Is the kill-capture ratio different in certain provinces? And is there a quantifiable difference between Generals McChrystal, Petraeus and Allen with regards to the capture‐or‐kill raids?
The report presents a wide array of findings, including the total number of reported deaths and detentions country-wide, their geographical distribution, their distribution over time, the ratio between deaths and detentions. The same information is then given for the deaths and detentions that were specifically reported as having taken place during capture-or-kill operations. An overview is given of the intended targets of the raids (as mentioned in the press releases), the average numbers of deaths and detention per raid and finally the total number of insurgent ‘leaders’ and ‘facilitators’ mentioned in the press releases is compared with the total aggregate figures that are periodically released. The report contains a large number of detailed figures and graphs. Accompanying maps and visualisations can be found here on the Guardian datablog website. The report finally concludes, among other things, that:
“ISAF may continue to hold that the capture‐or‐kill raids are the safest and most effective tool against the insurgency, but this remains to be proven, particularly in the context of the data cited in this report. The somewhat regular provision of data to media outlets is a positive gesture, but, as shown here, there is no way to evaluate the claims made, nor is it issued systematically enough to be able to draw even basic comparisons with previous months. The lack of transparency is particularly apparent in the case of the insurgent ‘leaders’ that were reportedly being killed and captured; there is no way to properly evaluate these claims, aside from in the manner that this report has attempted.”
The report, finally, ends with a set of suggestions on the kind of standardised data that would be highly useful if released to the media in future briefings.*
The core of ISAF’s response to the report is that its press releases were never intended to be an authoritative database of all operations, and that an analysis of them will for that reason inevitably result in ‘an overly simplistic, flawed and inaccurate product.’ It particularly feels that ‘the methodology of comparing data found in ISAF press releases and correlating that to battlefield effectiveness badly misrepresents the reality of ISAF operations’. It additionally reiterates its position on ‘night operations’ as one of the most effective ways of targeting key leaders and insurgents, with minimal collateral damage to the population.**
There seem to be two different issues here: First, the actual substance of the AAN report which, as described above, is careful to lay out the aims of the research, provides detailed descriptions of both the methodology and its shortcomings, and attempts to be fair and measured in its conclusions. Second, the media attention that surrounded the report’s launch – which focused, among others, on whether the report’s findings indicate that ISAF has exaggerated the success of its operations by inflating the numbers of insurgent leaders that were killed or captured (see for instance here andhere). The ISAF response seems to be mainly aimed at the latter. It does not engage with the substance of the report and it does not deal with the conclusions and recommendations that highlight the need for more systematic and detailed information to ensure a more measured and serious debate.
It is obvious that the dataset (consisting of 3,771 ISAF press releases) did not include all information available to ISAF and does not provide a full picture of all military operations in Afghanistan. This has been made clear throughout the report (there are up to thirty explicit individual statements, including a whole section, noting the incomplete nature of the data set). It was at the same time, however, equally clear that it was still highly useful to analyse the information that had been made available:
‘Although this data is not the complete picture (not all operations are written up as press releases), it offers insight into how ISAF sees its contribution to the war and presents a far more differentiated picture of the capture‐or‐kill raids than the released aggregate data on its own.’ Even though the information is not complete, it allows those studying it to ‘examine trends’ and to arrive at ‘some absolute minimum figures for the number of people claimed to have been killed or captured during the course of operations around the country’.
The fact that the press releases, in ISAF’s own words, ‘were never intended to be an authoritative database of all ISAF operations conducted in Afghanistan, nor even a representative sample’ does not preclude them from being a source of information and analysis – provided that the methodology is transparent and the authors are clear about the limitations of both the data and the conclusions.
ISAF’s response does however raise important questions. If the provided information is, in ISAF’s own words, ‘by design incomplete’, then it would be useful to clarify what the value of this information is. If ISAF press releases only ‘provide some details of insurgent casualties as part of its mandate to inform the public’, then it would be useful to know how decisions are made on what information is released and what kind of information is kept from the public. If the provided information cannot be considered ‘a representative sample’ of what may be worth knowing with regard to ISAF operations, then how should it be treated?*** The big underlying question is, if there are no ways of comparing or evaluating the information that is provided and the claims that are made, then what assurances are there that they can be taken at face value – particularly in the context of consistent and often not very sophisticated efforts to ‘shape the debate’ and to engage in ‘strategic communications’.
This is not about criticising ISAF. But in the swirl of statements and opinions there is simply not enough reliable information available in the public domain. AAN and the authors of the report have sought to provide an overview of the information that ISAF has so far released – which may or may not be an accurate representation of what occurs on the battle field. In doing so, they have sought to raise some key issues relating to ISAF’s communication and information management strategies. AAN is ready to continue this conversation in any format that might be useful.
* The report concludes that the current lack of information would be greatly helped by the release of standardised data on: the number of capture‐or‐kill raids, issued once a month and covering the entirety of each month; the number of individuals killed or captured during each month, issued once a month and covering the entirety of each month, ideally broken down by province; the number of individuals of those captured each month who were subsequently released; the proportion of raids that were initiated by ISAF commanders in the field as opposed to those initiated by ISAF headquarters (or elsewhere); the proportion of capture‐or‐kill operations each month that are not written up in official ISAF press releases; and clear definitions of all the terms used by ISAF in the press releases.
** ISAF states that: ‘Night operations, as addressed in the report, remain one of the most effective methods for targeting key leaders and insurgents — they minimize civilian casualties by conducting operations during a time when most civilians are not in and around targeted areas. Nearly 90 percent of the time, night operations are conducted without a shot being fired, and the intelligence based targeting is so precise that the potential for civilian casualties is quite low. During night operations, almost half the missions result in removing the intended target from the battlefield. ISAF operations are planned and conducted with the support of a vast set of intelligence to ensure the coalition and our Afghan partners are contributing to overall campaign progress, with minimal collateral damage to the population.
‘Night operations’ were incidentally not the main focus of the report, although capture-or-kill operations, which often take place at night, were. The report contains a whole section (section 2) which seeks to put the capture-or-kill operations in the context of all the reported incidents involving the death or detention of an individual.
*** The question whether ISAF has exaggerated the number of leaders and facilitators that were killed or captured remains open. ISAF’s response provides some explanation as to how the military understand the categories (see quote below) and reiterates that ISAF does not release information on all operations – given that this may ‘compromise sensitive information’ – but sidesteps the issue that there is no way for analysts or the wider public to evaluate, or properly interpret, the claims that are made.
‘Regarding confusion over leaders versus facilitators in the report, leaders are individuals responsible for providing direction to a group of people subordinate to them. Groups can vary in size, from less than 10, to dozens or even hundreds of individuals. Similar to other military units, insurgent leaders can have responsibility of leading formations ranging from a few individuals, to much larger numbers. Facilitators assist in the conduct of future operations by assisting insurgent movement, equipping, people, munitions, cash, etc. Individual roles can change and leaders, likewise, can absolutely be classified as facilitators.’
2011-10-CA-004
ISAF Responds to use of AAN News Releases Study
KABUL, Afghanistan (October 14, 2011) – ISAF takes serious issue with the misrepresentation of coalition operational information outlined in the “A Knock On the Door” report published by the Afghan Analysts Network (AAN). In general, the methodology of comparing data found in ISAF press releases and correlating that to battlefield effectiveness badly misrepresents the reality of ISAF operations and will confuse serious researchers or those engaged in balanced reporting on this subject. A careful read of the research methodology can easily lead to a number of faulty conclusions. ISAF was never consulted in the preparation of this study, nor asked for data regarding operations.
A flaw in the “analysis of press releases” is the lack of data in this report. When associated with the broad spectrum of combat operations and particularly special operations activities, it does not automatically result in a press release. Public disclosure, even after the fact, may result in compromise of sensitive information. Release of information in insurgent warfare is not always made public, so studies based on the use of press releases can be both incomplete and problematic.
There are dozens of special operations missions conducted across Afghanistan every week. The majority of these are partnered with and often led by Afghan forces. Most of these efforts are not detailed through news releases.
The report’s authors address night operations. ISAF press releases — the source of the author’s data — capture a wide variety of military operations conducted throughout the country at any time of the day. This data is not extrapolated out of the conclusions formulated by the authors.
Night operations, as addressed in the report, remain one of the most effective methods for targeting key leaders and insurgents — they minimize civilian casualties by conducting operations during a time when most civilians are not in and around targeted areas. Nearly 90 percent of the time, night operations are conducted without a shot being fired, and the intelligence based targeting is so precise that the potential for civilian casualties is quite low. During night operations, almost half the missions result in removing the intended target from the battlefield. ISAF operations are planned and conducted with the support of a vast set of intelligence to ensure the coalition and our Afghan partners are contributing to overall campaign progress, with minimal collateral damage to the population.
Regarding confusion over leaders versus facilitators in the report, leaders are individuals responsible for providing direction to a group of people subordinate to them. Groups can vary in size, from less than 10, to dozens or even hundreds of individuals. Similar to other military units, insurgent leaders can have responsibility of leading formations ranging from a few individuals, to much larger numbers.
Facilitators assist in the conduct of future operations by assisting insurgent movement, equipping, people, munitions, cash, etc. Individual roles can change and leaders, likewise, can absolutely be classified as facilitators.
The published ISAF press releases used in preparing the AAN report were never intended to be an authoritative database of all ISAF operations conducted in Afghanistan, nor even a representative sample from which to draw scholarly conclusions. Any analysis of complex combat operations based on press releases alone, which by definition are written to provide basic, factual information, inevitably will produce an overly simplistic, flawed and inaccurate product.
The analysis conducted in the AAN report lacks the broader context of the ISAF mission and the dynamics of the insurgency in Afghanistan. The nature of the insurgency, including its geographic focus, changed during the period Dec 2009 – Sep 2011. Overall violence levels are trending downward in 2011 and enemy initiated attacks are also down across the country. ISAF and Afghan National Security Force activity varies in intensity and location in response to the demands of the mission and conditions on the ground. Linking all these variations with other factors such as a change in ISAF command leadership to draw an operational conclusion is puzzling and completely spurious.
ISAF does not recognize insurgents killed or captured as the sole indicator of progress in the mission overall. ISAF press releases provide some details of insurgent casualties as part of its mandate to inform the public, but trying to create an analysis primarily using insurgents reported as “killed or captured” through press releases will inevitably lead to faulty conclusions and inaccurate data and in no way represents any type of analytical sampling.
Authoritative research cannot be conducted through mere analysis of press releases, since the release of information through such releases is, by design, incomplete. ISAF hopes this clarifies some of the great confusion generated by this study and welcomes rigorous analysis methodology in future endeavors.
ISAF Public Affairs Office
ISAF has taken ‘serious issue’ with AAN’s latest report ‘A Knock on the Door. 22 Months of ISAF Press Releases’ (see full text of the press release below). ISAF argues that the methodology is flawed and based on incomplete information, and it worries that this will ‘confuse serious researchers or those engaged in balanced reporting on this subject.’ The response, and the argument ISAF has with the AAN report, illustrates some of the key issues surrounding the attempts at information management and strategic communications by ISAF and others.
The AAN report, by Kandahar-based researchers Alex Strick van Linschoten and Felix Kuehn, analyses all information surrounding deaths and detentions during military operations contained in the ISAF press releases from 1 December 2009 to 30 September 2011. The report then zooms in on the deaths and detentions during the so-called capture-or-kill operations and compares this information, in particular with regard to the number of ‘leaders’ and ‘facilitators’, with the total figures that are periodically released.
The main questions the report sought to answer, based on the available information, were these: How many raids are taking place? How many people are being detained and how many are being killed? How are operations distributed geographically and over time? Is the kill-capture ratio different in certain provinces? And is there a quantifiable difference between Generals McChrystal, Petraeus and Allen with regards to the capture‐or‐kill raids?
The report presents a wide array of findings, including the total number of reported deaths and detentions country-wide, their geographical distribution, their distribution over time, the ratio between deaths and detentions. The same information is then given for the deaths and detentions that were specifically reported as having taken place during capture-or-kill operations. An overview is given of the intended targets of the raids (as mentioned in the press releases), the average numbers of deaths and detention per raid and finally the total number of insurgent ‘leaders’ and ‘facilitators’ mentioned in the press releases is compared with the total aggregate figures that are periodically released. The report contains a large number of detailed figures and graphs. Accompanying maps and visualisations can be found here on the Guardian datablog website. The report finally concludes, among other things, that:
“ISAF may continue to hold that the capture‐or‐kill raids are the safest and most effective tool against the insurgency, but this remains to be proven, particularly in the context of the data cited in this report. The somewhat regular provision of data to media outlets is a positive gesture, but, as shown here, there is no way to evaluate the claims made, nor is it issued systematically enough to be able to draw even basic comparisons with previous months. The lack of transparency is particularly apparent in the case of the insurgent ‘leaders’ that were reportedly being killed and captured; there is no way to properly evaluate these claims, aside from in the manner that this report has attempted.”
The report, finally, ends with a set of suggestions on the kind of standardised data that would be highly useful if released to the media in future briefings.*
The core of ISAF’s response to the report is that its press releases were never intended to be an authoritative database of all operations, and that an analysis of them will for that reason inevitably result in ‘an overly simplistic, flawed and inaccurate product.’ It particularly feels that ‘the methodology of comparing data found in ISAF press releases and correlating that to battlefield effectiveness badly misrepresents the reality of ISAF operations’. It additionally reiterates its position on ‘night operations’ as one of the most effective ways of targeting key leaders and insurgents, with minimal collateral damage to the population.**
There seem to be two different issues here: First, the actual substance of the AAN report which, as described above, is careful to lay out the aims of the research, provides detailed descriptions of both the methodology and its shortcomings, and attempts to be fair and measured in its conclusions. Second, the media attention that surrounded the report’s launch – which focused, among others, on whether the report’s findings indicate that ISAF has exaggerated the success of its operations by inflating the numbers of insurgent leaders that were killed or captured (see for instance here andhere). The ISAF response seems to be mainly aimed at the latter. It does not engage with the substance of the report and it does not deal with the conclusions and recommendations that highlight the need for more systematic and detailed information to ensure a more measured and serious debate.
It is obvious that the dataset (consisting of 3,771 ISAF press releases) did not include all information available to ISAF and does not provide a full picture of all military operations in Afghanistan. This has been made clear throughout the report (there are up to thirty explicit individual statements, including a whole section, noting the incomplete nature of the data set). It was at the same time, however, equally clear that it was still highly useful to analyse the information that had been made available:
‘Although this data is not the complete picture (not all operations are written up as press releases), it offers insight into how ISAF sees its contribution to the war and presents a far more differentiated picture of the capture‐or‐kill raids than the released aggregate data on its own.’ Even though the information is not complete, it allows those studying it to ‘examine trends’ and to arrive at ‘some absolute minimum figures for the number of people claimed to have been killed or captured during the course of operations around the country’.
The fact that the press releases, in ISAF’s own words, ‘were never intended to be an authoritative database of all ISAF operations conducted in Afghanistan, nor even a representative sample’ does not preclude them from being a source of information and analysis – provided that the methodology is transparent and the authors are clear about the limitations of both the data and the conclusions.
ISAF’s response does however raise important questions. If the provided information is, in ISAF’s own words, ‘by design incomplete’, then it would be useful to clarify what the value of this information is. If ISAF press releases only ‘provide some details of insurgent casualties as part of its mandate to inform the public’, then it would be useful to know how decisions are made on what information is released and what kind of information is kept from the public. If the provided information cannot be considered ‘a representative sample’ of what may be worth knowing with regard to ISAF operations, then how should it be treated?*** The big underlying question is, if there are no ways of comparing or evaluating the information that is provided and the claims that are made, then what assurances are there that they can be taken at face value – particularly in the context of consistent and often not very sophisticated efforts to ‘shape the debate’ and to engage in ‘strategic communications’.
This is not about criticising ISAF. But in the swirl of statements and opinions there is simply not enough reliable information available in the public domain. AAN and the authors of the report have sought to provide an overview of the information that ISAF has so far released – which may or may not be an accurate representation of what occurs on the battle field. In doing so, they have sought to raise some key issues relating to ISAF’s communication and information management strategies. AAN is ready to continue this conversation in any format that might be useful.
* The report concludes that the current lack of information would be greatly helped by the release of standardised data on: the number of capture‐or‐kill raids, issued once a month and covering the entirety of each month; the number of individuals killed or captured during each month, issued once a month and covering the entirety of each month, ideally broken down by province; the number of individuals of those captured each month who were subsequently released; the proportion of raids that were initiated by ISAF commanders in the field as opposed to those initiated by ISAF headquarters (or elsewhere); the proportion of capture‐or‐kill operations each month that are not written up in official ISAF press releases; and clear definitions of all the terms used by ISAF in the press releases.
** ISAF states that: ‘Night operations, as addressed in the report, remain one of the most effective methods for targeting key leaders and insurgents — they minimize civilian casualties by conducting operations during a time when most civilians are not in and around targeted areas. Nearly 90 percent of the time, night operations are conducted without a shot being fired, and the intelligence based targeting is so precise that the potential for civilian casualties is quite low. During night operations, almost half the missions result in removing the intended target from the battlefield. ISAF operations are planned and conducted with the support of a vast set of intelligence to ensure the coalition and our Afghan partners are contributing to overall campaign progress, with minimal collateral damage to the population.
‘Night operations’ were incidentally not the main focus of the report, although capture-or-kill operations, which often take place at night, were. The report contains a whole section (section 2) which seeks to put the capture-or-kill operations in the context of all the reported incidents involving the death or detention of an individual.
*** The question whether ISAF has exaggerated the number of leaders and facilitators that were killed or captured remains open. ISAF’s response provides some explanation as to how the military understand the categories (see quote below) and reiterates that ISAF does not release information on all operations – given that this may ‘compromise sensitive information’ – but sidesteps the issue that there is no way for analysts or the wider public to evaluate, or properly interpret, the claims that are made.
‘Regarding confusion over leaders versus facilitators in the report, leaders are individuals responsible for providing direction to a group of people subordinate to them. Groups can vary in size, from less than 10, to dozens or even hundreds of individuals. Similar to other military units, insurgent leaders can have responsibility of leading formations ranging from a few individuals, to much larger numbers. Facilitators assist in the conduct of future operations by assisting insurgent movement, equipping, people, munitions, cash, etc. Individual roles can change and leaders, likewise, can absolutely be classified as facilitators.’
2011-10-CA-004
ISAF Responds to use of AAN News Releases Study
KABUL, Afghanistan (October 14, 2011) – ISAF takes serious issue with the misrepresentation of coalition operational information outlined in the “A Knock On the Door” report published by the Afghan Analysts Network (AAN). In general, the methodology of comparing data found in ISAF press releases and correlating that to battlefield effectiveness badly misrepresents the reality of ISAF operations and will confuse serious researchers or those engaged in balanced reporting on this subject. A careful read of the research methodology can easily lead to a number of faulty conclusions. ISAF was never consulted in the preparation of this study, nor asked for data regarding operations.
A flaw in the “analysis of press releases” is the lack of data in this report. When associated with the broad spectrum of combat operations and particularly special operations activities, it does not automatically result in a press release. Public disclosure, even after the fact, may result in compromise of sensitive information. Release of information in insurgent warfare is not always made public, so studies based on the use of press releases can be both incomplete and problematic.
There are dozens of special operations missions conducted across Afghanistan every week. The majority of these are partnered with and often led by Afghan forces. Most of these efforts are not detailed through news releases.
The report’s authors address night operations. ISAF press releases — the source of the author’s data — capture a wide variety of military operations conducted throughout the country at any time of the day. This data is not extrapolated out of the conclusions formulated by the authors.
Night operations, as addressed in the report, remain one of the most effective methods for targeting key leaders and insurgents — they minimize civilian casualties by conducting operations during a time when most civilians are not in and around targeted areas. Nearly 90 percent of the time, night operations are conducted without a shot being fired, and the intelligence based targeting is so precise that the potential for civilian casualties is quite low. During night operations, almost half the missions result in removing the intended target from the battlefield. ISAF operations are planned and conducted with the support of a vast set of intelligence to ensure the coalition and our Afghan partners are contributing to overall campaign progress, with minimal collateral damage to the population.
Regarding confusion over leaders versus facilitators in the report, leaders are individuals responsible for providing direction to a group of people subordinate to them. Groups can vary in size, from less than 10, to dozens or even hundreds of individuals. Similar to other military units, insurgent leaders can have responsibility of leading formations ranging from a few individuals, to much larger numbers.
Facilitators assist in the conduct of future operations by assisting insurgent movement, equipping, people, munitions, cash, etc. Individual roles can change and leaders, likewise, can absolutely be classified as facilitators.
The published ISAF press releases used in preparing the AAN report were never intended to be an authoritative database of all ISAF operations conducted in Afghanistan, nor even a representative sample from which to draw scholarly conclusions. Any analysis of complex combat operations based on press releases alone, which by definition are written to provide basic, factual information, inevitably will produce an overly simplistic, flawed and inaccurate product.
The analysis conducted in the AAN report lacks the broader context of the ISAF mission and the dynamics of the insurgency in Afghanistan. The nature of the insurgency, including its geographic focus, changed during the period Dec 2009 – Sep 2011. Overall violence levels are trending downward in 2011 and enemy initiated attacks are also down across the country. ISAF and Afghan National Security Force activity varies in intensity and location in response to the demands of the mission and conditions on the ground. Linking all these variations with other factors such as a change in ISAF command leadership to draw an operational conclusion is puzzling and completely spurious.
ISAF does not recognize insurgents killed or captured as the sole indicator of progress in the mission overall. ISAF press releases provide some details of insurgent casualties as part of its mandate to inform the public, but trying to create an analysis primarily using insurgents reported as “killed or captured” through press releases will inevitably lead to faulty conclusions and inaccurate data and in no way represents any type of analytical sampling.
Authoritative research cannot be conducted through mere analysis of press releases, since the release of information through such releases is, by design, incomplete. ISAF hopes this clarifies some of the great confusion generated by this study and welcomes rigorous analysis methodology in future endeavors.
ISAF Public Affairs Office
Revisions:
This article was last updated on 9 Mar 2020
Tags:
ISAF
Taleban
