The Central Intelligence Agency's Engagement with Technology
Introduction
The Central Intelligence Agency (CIA), in conjunction with the broader United States intelligence community, has historically engaged with and significantly influenced the development and application of advanced technology, particularly in the realm of computing and information systems.
This involvement has spanned from foundational research to the shaping of commercial enterprises, driven by the imperative to enhance intelligence gathering, national security, and counter-terrorism capabilities. The collaboration between government agencies and the private sector, notably Silicon Valley, has enabled a comprehensive public-private mass surveillance state.
Origins of Intelligence Community Technological Investment
A long history exists of collaboration between America's leading scientists and the intelligence community, evident in projects ranging from the creation of Atomic Weapons and satellite technology to efforts associated with lunar exploration.
The internet itself originated from an intelligence effort; in the 1970s, the Defense Advanced Research Projects Agency (DARPA) linked four supercomputers to manage massive data transfers. This operation was later transferred to the National Science Foundation (NSF), which expanded the network across universities and eventually to the public, thereby establishing the architecture and scaffolding of the World Wide Web.
By the mid-1990s, the intelligence community recognised a significant opportunity as supercomputing capabilities began to transition from academic institutions to the private sector, spearheaded by investments in Silicon Valley.
A digital revolution was unfolding, promising to transform data gathering and the interpretation of vast information volumes. The intelligence community aimed to influence Silicon Valley's supercomputing initiatives from their inception to ensure their utility for both military and homeland security objectives.
The Central Intelligence Agency and the National Security Agency (NSA) understood that their future capabilities would likely be profoundly shaped outside government institutions, especially as military and intelligence budgets faced constraints within the Clinton administration, while the private sector commanded extensive resources. Conducting mass surveillance for national security purposes necessitated cooperation between the government and emerging supercomputing companies.
The Massive Digital Data Systems (MDDS) Project
To achieve these objectives, the intelligence community initiated engagement with scientists at American universities who were at the forefront of the supercomputing revolution. These scientists were developing methods capable of gathering and intelligently interpreting immense quantities of data, a feat beyond the scope of individual human analysts at the NSA and CIA.
In the mid-1990s, the intelligence community provided funding to promising supercomputing endeavours across academia, guiding the development of systems designed to make vast amounts of information beneficial for both the private sector and intelligence operations.
This funding was channelled through an unclassified, highly compartmentalised programme known as the Massive Digital Data Systems (MDDS) project, which was managed for the CIA and NSA by prominent military and intelligence contractors.
The MDDS project was introduced to dozens of leading computer scientists at institutions such as Stanford, CalTech, MIT, Carnegie Mellon, and Harvard, through a white paper outlining the aspirations of the CIA, NSA, DARPA, and other agencies. The research was predominantly funded and managed by unclassified science agencies like the NSF, facilitating its scalability within the private sector should it meet the intelligence community's expectations.
A 1993 MDDS white paper from the intelligence community stated, Not only are activities becoming more complex, but changing demands require that the Intelligence Community process different types as well as larger volumes of data. Consequently, the Intelligence Community is taking a proactive role in stimulating research in the efficient management of massive databases and ensuring that IC requirements can be incorporated or adapted into commercial products. The Community Management Staff (CMS) commissioned an MDDS Working Group to address these needs and identify possible solutions.
Over several years, the programme intended to award more than a dozen grants, each worth several million pounds, to advance this research. These grants were primarily directed through the NSF to ensure that successful efforts could be protected as intellectual property, forming the basis for companies that would attract investment from Silicon Valley.
This model of public-to-private innovation spurred the creation of influential science and technology companies, including Qualcomm, Symantec, and Netscape, and supported crucial research in fields such as Doppler radar and fibre optics, which are fundamental to modern corporations like AccuWeather, Verizon, and AT&T. The NSF presently supplies almost 90% of all federal funding for university-based computer-science research.
The Vision for Digital Tracking
The research divisions of the CIA and NSA sought to enable academic computer scientists to identify what they termed birds of a feather. The premise was that just as geese fly in formations or sparrows move in unison, like-minded groups of humans would exhibit collective behaviour online.
The intelligence community's inaugural unclassified briefing for scientists, held in San Jose in the spring of 1995, was titled the Birds of a Feather Session on the Intelligence Community Initiative in Massive Digital Data Systems.
The central aim of this research was to track digital fingerprints within the rapidly expanding global information network, then known as the World Wide Web. The intelligence community posed questions regarding the organisation of an entire world of digital information to allow human requests within the network to be tracked and sorted, for queries to be linked and ranked by importance, and for birds of a feather to be identified to facilitate organised tracking of communities and groups.
By collaborating with emerging commercial-data companies, the intent was to track such groups across the internet and identify them via their digital footprints, akin to forensic scientists using smudges to identify criminals.
The prediction was that potential terrorists, like birds of a feather flocking together, would communicate within this new global network, and their patterns could be discerned from the vast influx of new information. Once identified, their digital trails could be followed comprehensively.
Google's Covert Origins
In 1995, one of the earliest and most promising MDDS grants was awarded to a computer-science research team at Stanford University, which had a decade-long history of working with NSF and DARPA grants.
The primary objective of this grant was query optimisation of very complex queries that are described using the query flocks approach. A second grant, a DARPA-NSF grant closely associated with Google's genesis, formed part of a coordinated effort to construct a massive digital library utilising the internet as its foundation.
Both grants funded the research of two Jewish graduate students, Sergey Brin and Larry Page, who were making rapid advancements in web-page ranking and the tracking and interpretation of user queries.
The research conducted by Brin and Page under these grants became the core technology of Google: individuals employing search functions to locate specific information within extensive data sets. However, the intelligence community recognised a different utility in their research: the potential for the network to be organised with such efficiency that individual users could be uniquely identified and tracked.
This process is ideally suited for counter-terrorism and homeland security efforts, enabling the unique identification of human beings and like-minded groups who might pose a threat to national security online before they inflict harm.
This utility explains the intelligence community's strong interest in Brin and Page's research, as the CIA had previously relied predominantly on human intelligence efforts in the field for identifying threats; the capability to track them virtually, in conjunction with field operations, would fundamentally alter intelligence methodologies.
This marked the nascent stage of what would become Google within a few years. The two intelligence community managers overseeing the MDDS program met regularly with Brin as his research progressed, and he contributed as an author to several research papers arising from this MDDS grant before he and Page departed to establish Google. Brin functioned as a Stanford researcher, utilising grants provided by the NSA and CIA through the unclassified MDDS program.
Despite its pivotal role, the MDDS research effort has never been formally incorporated into Google's public origin narrative. The principal investigator for the MDDS grant explicitly stated that Google directly resulted from their research, asserting that Its core technology, which allows it to find pages far more accurately than other search engines, was partially supported by this grant. A published research paper featuring Brin's seminal work also references the NSF grant generated by the MDDS program.
Instead, Google's origin stories typically acknowledge only one federal grant: the NSF/DARPA digital libraries grant, which facilitated Stanford researchers' ability to search the entire World Wide Web hosted on the university's servers. Stanford's Infolab attributes Google's development to this funding, stating, The development of the Google algorithms was carried on a variety of computers, mainly provided by the NSF-DARPA-NASA-funded Digital Library project at Stanford.
Similarly, the NSF's historical account of Google's origins mentions only the digital libraries grant, omitting the MDDS grant. In their renowned research paper, The Anatomy of a Large-Scale Hypertextual Web Search Engine, Brin and Page expressed gratitude to the NSF and DARPA for their digital library grant to Stanford.
However, the grant from the intelligence community’s MDDS program, specifically conceived for the breakthrough upon which Google was founded, has largely become obscure.
In 2006, Google publicly refuted claims of receiving funding from the intelligence community for counter-terrorism efforts, stating the statements related to Google are completely untrue. While the CIA did not directly fund or create Google, Brin and Page's research, supported by these grants, precisely aligned with the objectives of the NSA, CIA, and the wider intelligence community.
The CIA and NSA funded an unclassified, compartmentalised programme specifically designed from its inception to foster the development of a system almost identical to Google. Brin’s groundbreaking research on page ranking through user query tracking and search linking, effectively identifying birds of a feather, was a central aim of the intelligence community's MDDS program, and Google's success surpassed their most ambitious projections.
The Intelligence Community's Enduring Legacy in Silicon Valley
Digital privacy concerns have intensified regarding the intersection between the intelligence community and major commercial technology corporations. The extent to which the intelligence community relies on the world’s largest science and technology companies for its counter-terrorism and national security operations remains largely unrecognised by the public.
Civil-liberty advocacy groups have expressed privacy concerns for many years, particularly concerning the Patriot Act. Enacted just 45 days after the September 11th Attacks under the guise of national security, the Patriot Act represented one of many legislative changes that facilitated government surveillance of ordinary Americans by broadening the authority to monitor phone and email communications, collect financial records, and track the internet activity of innocent citizens. Although widely perceived as a tool to apprehend terrorists, the Patriot Act in practice designates ordinary citizens as suspects.
Leading technology and communications companies, including Verizon, AT&T, Google, Facebook, and Microsoft, assert that they do not proactively or deliberately provide their extensive customer databases to federal security and law enforcement agencies. They maintain that they only comply with subpoenas or requests that are properly filed under the provisions of the Patriot Act. Nevertheless, an examination of recent public records indicates a continuous stream of requests that could potentially undermine the spirit of this privacy commitment.
Between 2016 and 2017, governmental authorities at local, state, and federal levels issued over 260,000 subpoenas, court orders, warrants, and other legal requests to Verizon, more than 250,000 to AT&T, and nearly 24,000 to Google, all seeking information related to national security, counter-terrorism, or criminal investigations.
While direct national security or counter-terrorism requests constitute a minor proportion of these totals, the legal process established by the Patriot Act has become so routine that these companies maintain dedicated employee teams to manage the constant influx of requests.
This collaboration between the intelligence community and major commercial science and technology companies has been exceptionally successful. When national security agencies require the identification and tracking of individuals and groups, they are aware of where to seek assistance and do so frequently. This outcome aligns with the initial objectives and has succeeded perhaps more extensively than anyone could have envisioned at the time.