Please note: In this problem, there are two fictional countries: Ul’Sanda and Rinia. These two countries are not at war, but each is known to carefully monitor the other.
Situation:
You and your Ul’Sandan investigative team have been tasked with assessing whether a temporary Ul’Sandan Black Site (unacknowledged military facility) was observed by foreign surveillance.
The Black Site was located in the middle of a small island off the coast of Ul’Sanda, not visible from the sea. The island is small, remote, and unpopulated; it can safely be assumed that any potential surveillance of the island was airborne.
Crucially, the Black Site team had lookouts stationed at it throughout the day. One of the lookouts reported seeing (and then photographing) some thin, wispy clouds which looked like potential contrails from an aircraft. This raised alarms due to the possibility of a surveillance aircraft having flown overhead. Apart from this, there were no other lookout observations indicating that an aircraft of any sort overflew the island. Of course, there is the possibility that the lookouts missed another aircraft flying over the site.
The site was located on the island for approximately 24 hours, before being moved elsewhere. It is your job to determine the probability that the site was overflown by aerial reconnaissance during that time.
There are a number of possible candidates that may have flown over the island over the period of 24 hours. The two leading candidates, based on the location of the island, and known historical flight paths, are:
- Rinian Spy Plane: These unmanned aircraft have been known to fly over theareas close to the Ul’Sandan mainland. When the site was moved onto theisland, Ul’Sandan intelligence - based on evidence available at the time(including known flight paths) - concluded that there was a 10% probabilitythat a Rinian spy plane would fly over the island during that 24-hour period. Inthe intelligence community, a 10% probability is taken to mean that an assessed event is highly improbable.
- Ul’Sandan Drone: Operated by the Ul’Sandan Navy and Air Force, these remote operated aircraft have long-range capabilities, and have been known to follow flight paths that take them over the island. When the site was moved onto the island, Ul’Sandan intelligence - based on evidence available at the time (including known flight paths) - concluded that there was a 5% probability that a Ul’Sandan Drone would fly over the island during that 24-hour period. In the intelligence community, a 5% probability is taken to mean that an assessed event is remote.
Questions:
For your answers, assume the facts to be as stated in the problem.
- What do you believe is the probability of a Rinian spy plane having overflown the Black Site, referring to all relevant evidence?
- What do you believe is the probability of a Ul’Sandan drone having overflown the Black Site, referring to all relevant evidence?
- Which of these two scenarios (Rinian spy plane or Ul’Sandan drone flying over the island) is more likely?
- Does the report need to recommend notifying counter-intelligence agencies immediately (i.e. is the probability of the Black Site having being overflown by a Rinian spy plane over 50%, having considered the evidence)?
Please provide your reasoning for all responses, referring to the evidence in each case.
List of evidence sources:
You are tasked with evaluating the impact of reports from the following sources:
- Foreign Intelligence Intercepts of Rinian Spy Plane activity
- Ul’Sandan Department of Defence report on Ul’Sandan Drone activit
- Domestic Contrail Expert on Rinian spy planes
- Domestic Contrail Expert on Ul’Sandan Drones
- Radar Station (Technical Report)
Critical purpose of investigation:
There is a pertinent purpose for your investigation: Crucially, if – having evaluated the evidence available to you – you believe the probability of a Rinian spy plane having flown over the island during the 24-hours that the Black site was on the island surpasses 50% (even-odds)
by any amount, then
you are required to recommend notifying counter-intelligence agencies immediately. If you do not think this probability exceeds 50%, immediate notification is not required and you can postpone communicating your judgment - one way or the other - for up to 48 hours on the possibility that more information may come available.
Rinian Spy Plane Evidence
Evidence 1:
Foreign Intelligence Intercepts
Translated intercepts of Rinian intelligence activities. These are translated, assessed, and summarised by experts on Rinian intelligence and counterintelligence. Due to the possibility of mistranslations, misinterpretations, and the possible sowing of false information, the following statistics on how to consider the uncertainty surrounding reports have been compiled for your use:
Report Reliability Statistics:
False Positive Rate (reporting that activity occurred, when in fact none took place):10%
False Negative Rate (reporting that activity did not occur, when in fact it did take place): 15%
Report Summary:
The report states that, based on intercepted Rinian intelligence transmissions, a Rinian spy plane flew over the island while the Black Site was present.
Rinian Spy Plane Evidence
Evidence 2:
Domestic Contrail Expert on Rinian Spy Planes
A member of your team is an expert on the contrails of Rinian spy planes, and has experience with a large number of prior incidents, so you show the expert the image taken of the potential contrails.
Your teammate’s history of reliability and accuracy reveals, however, that his assessments are not perfect. In particular, verification statistics indicate that this expert is overly bold; frequently proclaiming that suspicious cloud formations /contrails are the contrails of Rinian spy planes when they in fact are not (although he sometimes makes the opposite mistake). Checking the expert’s verified performance statistics you find:
Expert Reliability Statistics:
False Positive Rate (classifying suspicious image as a Rinian spy plane contrail, when in fact it is not): 40%
False Negative Rate (classifying suspicious image as not a Rinian spy plane contrail, when in fact it is): 20%
Report Summary:
The expert reports that, based on his assessment of the image, a Rinian spy plane was not flying over the island at the time the images were captured.
Ul’Sandan Drone Evidence
Evidence 3:
Department of Defense Report
As an Ul’Sandan official, you have access to Ul’Sandan military logs regarding possible flight paths of Ul’Sandan aircraft – including Ul’Sandan drones. These reports can generally be considered to be highly reliable, and the report reliability statistics are taken to be the following:
Report Reliability Statistics:
False Positive Rate (reporting that a certain flight path was followed when in fact it was not): 5%
False Negative Rate (reporting that a certain flight path was not followed when in fact it was the path taken): 5%
Report Summary:
The report states that, based on an assessment of Defence Department logs, a Ul’Sandan drone flew over the island while the Black Site was present.
Ul’Sandan Drone Evidence
Evidence 4:
Domestic Contrail Expert on Ul’Sandan Drones
Contrails of Ul’Sandan drones are quite different from contrails of Rinian spy planes. A member of your team is an expert on the contrails of Ul’Sandan drones, and has experience with a large number of prior incidents, so you show the expert the image taken of the potential contrails.
Your teammate’s history of reliability and accuracy reveals, however, that her assessments are not perfect. In particular, verification statistics indicate that this expert is overly cautious; frequently proclaiming that suspicious cloud formations / contrails are not the contrails of Ul’Sandan Drones when they in fact are (although she sometimes makes the opposite mistake). Checking the expert’s verified performance statistics you find:
Expert Reliability Statistics:
False Positive Rate (classifying suspicious image as an Ul’Sandan drone contrail, when in fact it is not): 20%
False Negative Rate (classifying suspicious image as not an Ul’Sandan drone contrail, when in fact it is): 30%
Report Summary:
The expert reports that, based on her assessment of the image, a Ul’Sandan drone was not flying over the island at the time the images were captured.
Technical Report Evidence
Evidence 5:
Radar Station automated logs
An old radar listening station is located on the island, covering the area of sky above the island’s landmass.
These stations were automated, and were designed to look for particular radar signatures, providing a simple daily report log that indicated positive if anything meeting its target signature characteristics had been detected in the last 24 hours(and negative if nothing had).
Automated Report Log Reliability:
Even now, the automated report logs are considered highly reliable, in that they have very low chances of false positives (0.5%) if no such signatures were present. There is however, an issue with the report logs: Although the radar station was originally designed to look for signatures corresponding to the typical size and altitudes of Rinian spy planes, in more modern times, two types of plane are known to fit these criteria: Rinian spy planes and Ul’Sandan drones. As such, the engineers who maintain the station provide you with the statistics they have regarding typical detection probabilities for each of these two cases:
Ul-Sandan drones have typically triggered positive reports 95% of the time they have been known to fly within the range of the radar, while modern Rinian spy planes have typically triggered positive reports 90% of the time that they are known to be within the area covered by the radar.
Automated Report Log:
The report comes back as positive for the 24-hour period when the Black Site was present.
