Recently the Office of the Director of National Intelligence (ODNI) unveiled a new strategy for open-source intelligence (OSINT) and referred to OSINT as the “INT of first resort”. Public and private sector organizations are realizing the value that the discipline can provide but are also finding that the exponential growth of digital data in recent years has overwhelmed many traditional OSINT methods. Thankfully, Artificial Intelligence (AI) and Machine Learning (ML) are starting to provide a transformative impact on the future of information gathering and analysis.
What is Open-Source Intelligence (OSINT)?
Open-Source Intelligence refers to the collection and analysis of information from publicly available sources. These sources can include traditional media, social media platforms, academic publications, government reports, and any other data that is openly accessible. The key characteristic of OSINT is that it does not involve covert or clandestine methods of information gathering such as human intelligence or social engineering. If I could have obtained data during my time working for the U.S. Government but I no longer can as a civilian, that isn’t OSINT.
Historically, OSINT has been a labor-intensive process involving several key steps:
- Identification of sources: Analysts determine which public sources are likely to contain relevant information.
- Data collection: Information is gathered from these sources, often through manual searches or web scraping tools.
- Data processing: The collected information is organized and structured for analysis.
- Analysis: Skilled analysts examine the data to identify patterns, trends, and insights.
- Reporting: Findings are compiled into reports for decision-makers to enable more informed decisions.
While effective, this approach faces limitations with the sheer volume of information available. Human analysts struggle to process everything manually and valuable insights may be hidden in complex patterns that are difficult for humans to detect. This is where AI/ML can provide a tremendous benefit in how information can be collected, processed and analyzed, thus freeing the human analyst to focus on things they are uniquely qualified for such as providing context. As a side benefit, this shift often improves morale as humans spend less time on mundane processing tasks and more time analyzing and reviewing information.
Tasks where AI/ML can provide immediate benefit include:
- Handling Massive Data Volumes: AI systems can process and analyze enormous amounts of data at speeds far beyond human capabilities. This allows OSINT practitioners to cast a much wider net than previously possible and still deal with the results.
- Real-time Analysis: The volume of information flow in today’s digital world is staggering. AI-powered OSINT tools can monitor and analyze data streams in real-time, providing up-to-the-minute intelligence and enabling rapid response to emerging situations.
- Multilingual and Multimodal Analysis: AI can break down language barriers by translating and analyzing content in multiple languages simultaneously. Moreover, it can process various data types – text, images, audio, and video – in an integrated manner, providing a more comprehensive intelligence picture. Many of these capabilities such as OpenAI’s Whisper can be utilized offline, thus removing any concerns about operational security (OPSEC).
- Predictive Analytics: By analyzing historical data and current trends, AI can help predict future events or behaviors, adding a proactive dimension to OSINT.
- Automation of Routine Tasks: AI can help automate many time-consuming aspects of OSINT, such as data collection and initial filtering, freeing human analysts to focus on higher-level analysis and decision-making. Things that were previously very difficult if not impossible to implement, such as accurate sentiment analysis, are now trivial.
At SANS Network Security the SEC497 Practical OSINT course and the SEC587 Advanced OSINT course will provide students with hands-on experience utilizing these AI capabilities to not only provide an increase in productivity, but also discover new possibilities.
While no technology is perfect, and we must consider the potential ramifications that a hallucination could cause before we implement AI, key pieces of technology currently being utilized for OSINT include:
- Natural Language Processing (NLP): NLP allows machines to understand, interpret, and generate human language. In OSINT, NLP is crucial for:
- Sentiment analysis of social media posts
- Entity recognition to identify people, organizations, and locations in text
- Topic modeling to categorize large volumes of text data
- Machine translation for multilingual intelligence gathering
- Computer Vision: This technology enables machines to interpret and analyze visual information. In OSINT, computer vision is used for:
- Facial recognition in images and videos
- Facial comparisons to identify if the same person is featured in multiple images
- Object detection in imagery
- Optical character recognition (OCR) to extract text from images
- Scene understanding in video footage
- Machine Learning and Data Mining: How many times have you heard “those who don’t know history are doomed to repeat it”? Machine Learning is the personification of that concept as it allows systems to learn from data and improve their performance over time. In OSINT, they are used for:
- Predictive analytics to forecast trends or events
- Anomaly detection to identify unusual patterns or behaviors
- Clustering and classification of data for easier analysis
- Network analysis to understand relationships between entities
I’ve been doing OSINT for almost two decades and this is by far the most dynamic, and exciting time I’ve seen with new developments in the space literally occurring daily. If you’re going to be at Network Security in Las Vegas this September, I look forward to discussing how this capability can improve our effectiveness and efficiency today, as well as what we can expect in the future.
Not yet registered for SANS Network Security? Check out this page to see all that’s in store!
Note: This article is expertly written by Matt Edmondson, a SANS Principal Instructor and Principal at Argelius Labs, with a decade of professional OSINT experience.