Can integrated Clear and Dark Web data revolutionise intelligence investigations?

The world of intelligence gathering has evolved dramatically. While infiltrating clandestine meetings in darkened rooms still has its place, today, a wealth of information resides online waiting to be unearthed and analysed. This blog post explores how investigators can leverage Clear and Dark Web data holistically together to gain critical insights and solve complex cases. 

A Familiar Landscape: Clear Web Investigations 

The Clear Web, the internet known to most that’s neatly indexed by standard search engines, is a treasure trove of publicly available, readily accessible information. Easy for investigators to search through, the Clear Web is a great starting point for building a comprehensive picture of a subject or situation for several critical intelligence investigation use cases: 

• Open Source Intelligence (OSINT) collects and analyses information from search engines, social media platforms, news sites, public records, and company websites to identify assets and connections between individuals, track movements, and establish timelines. 
• Social Media Analysis unlocks goldmines of personal information using social profiles, posts, photos, and connections to understand subjects’ interests, relationships, activities, and sentiment and help identify potential threats, track individuals, and understand group dynamics. 
• Media Monitoring helps track public sentiment and potential threats by looking at news articles, blog posts, and forum discussions to provide context and insights into events, individuals, organisations and cultural trends.  
• Background Checks to verify identities, uncover criminal histories, and identify financial connections using public records such as court records, property records, and business registrations for risk assessment and due diligence. 

The Deep and Dark Web: the hidden depths of the Internet 

This Clear Web, however, represents a tiny fraction of the Internet’s information. Over 95% of this content resides below the surface of the Clear Web, in what’s known as the Deep Web. At its most basic, anything behind a subscription, encryption or password, counts within this.  

For investigators needing deeper insights for more complex investigations, a sub-section of the Deep Web, the Dark Web, is a far more valuable, albeit challenging, information landscape.  

The Dark Web is a hidden part of the internet, accessible only via specialised browsers, often TOR (The Onion Router) browser, a modified, open-source version of Firefox. TOR anonymises web traffic using an encryption technique originally developed by the US Navy. It hides IP addresses and browsing activity by routing traffic through multiple nodes. This layered encryption ensures strong anonymity, protecting user privacy even if individual nodes are compromised. 

Most people perceive the Dark Web to be a place synonymous with illicit activities. And it’s true that illegal marketplaces and forums for drugs, weapons, stolen data, illegal pornography, counterfeits, Malware and other criminal activities exist there –c.57% of its activities according to 2020 research.  

However, the Dark Web also serves as a platform for secure communication and legal cryptocurrency trading, attracting whistleblowers, activists, and individuals seeking privacy, including those living under regimes with limited freedom of speech. The BBC, CIA and Facebook all have TOR sites on the Dark Web for this reason. Ultimately, the Dark Web’s anonymity, while exploited by criminals, makes it a valuable source of intelligence. 

A complex shifting world: the challenge of Dark Web Investigations 

The Dark Web is volatile in nature, with sites popping up and disappearing in rapid succession, making it difficult to get a precise view of how many sites there are and – due to the levels of anonymity – how many users there are too. Currently, it’s estimated there are over 2.7 million active daily Dark Web users  and it’s a mature and resilient space that continually adapts to site closures. 

To effectively use the Dark Web for intelligence, investigators need specialised tools, in-depth knowledge, refined techniques, and a keen awareness of ethical implications for these critical use cases. 

• Tracking Criminal Activity by monitoring illegal marketplaces to identify sellers, buyers, and track the flow of illegal goods. This is where effective Dark Web analysis tools are vital to help deanonymise individuals and generate intelligence to disrupt criminal networks.  
• Identifying Cyber Threats: Cybercriminals often discuss vulnerabilities and sell stolen data on Dark Web forums. Monitoring and carefully engaging in these forums can help investigators identify threats and prevent attacks. 
• Investigating Financial Crimes: Cryptocurrency transactions used in Dark Web marketplaces for legal and illegal trading – the most famous Bitcoin – can be difficult to trace. Investigators use specialised tools and techniques for blockchain data analysis to identify criminal individuals. 
• Uncovering Insider Threats: The Dark Web’s anonymity can embolden individuals to leak sensitive information. Investigators can monitor forums for leaked data and identify potential insider threats within organisations. 
• Sourcing Human Intelligence (HUMINT): While challenging, Intelligence investigators can establish contact with individuals who possess valuable information. Particularly useful for organised crime, terrorism, or other sensitive investigations. 

Challenges and Ethical Considerations in Clear and Dark Web investigations 

The Clear and Dark Web present both unique and shared investigation challenges requiring specialised skills, tools, and strategies: 

• Sheer volume of data on the Web makes it difficult to pinpoint relevant information. 
• Encryption of communications and transactions further complicates access to crucial evidence.  
• Crimes often span multiple jurisdictions, requiring national and international cooperation and collaboration. 
• Data fragmentation across various platforms and databases also requires extensive effort to piece together information.  
• Privacy laws and regulations add more complexity to obtaining data. Investigators must always operate within the bounds of the law, ensuring any intelligence collected can be used as admissible evidence in court.

The Dark Web has its own particular challenges: 

• Anonymity is the single most challenging factor which prevents linkages to real-world identities 
• Heavy encryption of transactions and communications further hinder interception and decoding of information, requiring specialist tool proficiency, cryptography and blockchain capabilities. 
• Human Analysis: while the sheer volume of Dark Web data necessitates using sophisticated tools to cut through the noise, careful analysis is vital to avoid false attributions. 

 Trends in the Evolving Investigation Landscape 

The world of online intelligence gathering is constantly evolving, requiring investigators to adapt their techniques accordingly: 

• Artificial Intelligence (AI) and Machine Learning technologies are both a challenge and opportunity to investigators. AI deepfake imagery, voice and video, AI-generated illicit content, cryptocurrency laundering and AI-automated cyberattacks, phishing and chatbots will require investigators to constantly adapt their techniques.  

On the flip side, AI can help automate the collection and analysis of vast amounts of data in forums and social channels, quickly identify patterns and anomalies, and predict future behaviour. AI facial recognition tooling was used to solve a recent joint Homeland Security Investigations (HIS) and UK police child exploitation case and in an HSI exploitation cold case review, resulting in hundreds of identifications of victims and perpetrators. 

• Big Data Analytics tools can process and analyse the exponentially growing large volumes of data, revealing hidden connections and potential insights about complex criminal networks or individuals’ motivations that would be impossible to detect manually. 

• Blockchain Analysis will be an even more critical skill for investigators given the growth of new cryptocurrencies like Monero (XMR) with highly advanced cryptographic techniques that mask transactions and dynamically change IP addresses, even as Bitcoin can now be ‘cracked’. With central banks also integrating cryptocurrency into operations, it’s clear its continuing adoption and acceptance for both legitimate and illicit transactions will remain a focus. 

• Decentralised Web (Web3), while slower to develop than predicted, just like AI presents both opportunities and challenges for investigators. Web3’s decentralisation, blockchain technology, and token-based economics, will require new tools and techniques to effectively investigate its platforms. 

• Focus on Privacy and Data Protection is an increasing challenge for investigators. New regulation like the UK’s incoming Data Protection and Digital Information Bill, Brazil’s General Personal Data Protection Act (LGPD) and India’s Personal Data Protection Bill mean investigators must be mindful of the latest legal and ethical frameworks they are operating under. Investigators must always adhere to such regulation and obtain proper warrants and authorisations before accessing sensitive information. 

The Integrated Approach: Combining Clear Web and Dark Web Intelligence 

The Clear Web and the Dark Web are both valuable sources of intelligence for investigators. The Clear Web offers a wealth of publicly available information, the Dark Web provides access to hidden data and insights that can be crucial to solve complex cases. By effectively combining intelligence from both realms and adapting to the increasingly complex technological landscape, investigators can gain a significant advantage in their pursuit of truth and justice.  

For example, several notable hackers and cyber-criminals have been arrested and subsequently jailed through integrating data from Clear and Deep web platforms like Roblox, Minecraft, Discord and Telegram with intelligence gathering on the Dark Web.  

This integrated Clear, Deep and Dark Web approach provides investigators with a broader, more nuanced understanding, yet the sheer volume, fragmentation and type of data means it’s a significant technical and practical challenge to navigate. It typically requires using multiple specialist tools and robust investigator skills, set against the dynamic nature of the Web itself. 

DarkBlue: a user-friendly platform for integrated Web intelligence investigations 

This fundamental challenge of scale, scope and complexity was the reason behind CACI developing our DarkBlue Intelligence suite.  

DarkBlue offers investigators a user-friendly, single OSINT platform to undertake holistic, complex investigations on the Clear, Deep and Dark Web efficiently, ethically and safely. 

DarkBlue leverages the intelligence that CACI has been scraping from across the Web including Tor, I2P, ZeroNet, OpenBazaar and Freenet for over 10 years, amassing billion of pages of data and capturing sites long since deleted.  

Included in the suite is DarkPursuit tool, which provides the user with a safe, anonymous browsing environment that obfuscates technical details that could be used for attribution or tracking. DarkPursuit integrates multiple specialist tools and allows investigators to seamlessly transition between search findings, multiple live environments and analysis.  

DarkPursuit’s new CluesAI feature helps investigators deanonymise individuals and entities on the Dark Web more efficiently, helping tackle its biggest – and growing – intelligence investigation challenge.  

CluesAI automatically gathers potentially identifying information like email addresses, cryptocurrency wallet details, and port scans from the Dark Web. It cross-references this information against DarkBlue’s extensive database and uses generative AI to identify connections and patterns. It then generates reports that summarise and highlight potentially deanonymising information, providing investigators with actionable leads in one click.  

As the Web in all its forms grows in complexity and size – particularly with the growth of Web3 and cryptocurrency, it’s vital that investigators can stay ahead of emerging threats to help protect national security and combat criminal activity.  

DarkBlue – and CACI’s OSINT as a Service offering – provides investigators with the critical tools and support from experienced intelligence experts to support your critical mission.

Contact us today to discuss how we can supercharge your investigations.