The Rise of Automated Fraud: Architecting Security Against "AI Slop"

Introduction to AI and Its Pervasive Role in Society

• The concept of Artificial Intelligence (AI) has been around for a long time, with the industry undergoing significant transformations over the last few decades, and particularly in the last few years, with AI being incorporated into various products and services 10s.
• The ubiquity of AI in society today is evident, with many companies advertising their products as AI-enabled, AI-boosted, or AI-infused, and even everyday items like toothbrushes being marketed as having artificial intelligence 2m6s.
• The use of AI has become so widespread that it is being used to generate content, and then again to summarize that content, creating a cycle of AI-generated information 2m6s.
• Despite the growing use of AI, there are concerns about its impact, with some statistics suggesting that AI use may drop significantly when it is not being used for its primary application, and others warning about the potential risks of AI infrastructure building 4m30s.

Understanding AI and Common Misconceptions

• The Financial Times has concluded that AI will either be humanity's salvation, destruction, or something in between, highlighting the uncertainty and debate surrounding the technology 6m30s.
• The term AI is often misunderstood, with many people conflating it with Artificial General Intelligence (AGI), and the concept of AI has been shaped by influential figures like Alan Turing, who published a seminal paper on Computing Machinery and Intelligence 8m40s.
• Turing's idea of the Turing test, which proposes the creation of a computer system that can imitate a human being, was initially met with skepticism, with some arguing that there is no practical reason to create machine intelligence indistinguishable from human intelligence 10m50s.
• People's understanding of artificial intelligence (AI) is often influenced by science fiction and stories, rather than technical sources, which can lead to confusion and misconceptions about the technology, particularly when it comes to advanced concepts like artificial general intelligence 10s.

Generative AI and Its Capabilities

• The Gell-Mann Amnesia effect is a cognitive bias that describes how people tend to be critical of information within their area of expertise, but assume that information outside of their expertise is accurate, making them more susceptible to being fooled by AI-generated content 2m6s.
• Generative AI has made significant progress in recent years, from simple applications like putting Nicolas Cage's face in unusual places to more sophisticated uses like creating realistic deepfakes, such as the Tom Cruise impersonations by Miles Fisher 4m30s.
• The launch of Sora by OpenAI has allowed users to create highly realistic videos, including ones that use the image and voice of public figures like Mark Cuban, who has given permission for his likeness to be used, but this has also raised concerns about copyright and the potential for misuse 6m40s.
• The importance of copyrighted content to generative AI is a significant issue, as models that rely on copyrighted material can produce more realistic results, but also raise legal and ethical concerns, as seen in the example of Midjourney and Adobe Firefly, which have different approaches to using copyrighted content 10m20s.
• The trade-off between using copyrighted content and avoiding it is a key challenge for generative AI models, with models that use copyrighted material often producing more realistic results, but also facing potential legal and ethical issues, as illustrated by the difference between Midjourney's and Adobe Firefly's responses to the prompt "Chewbacca reading" 12m50s.

AI-Generated Content and Its Impact on Jobs and Media

• The concept of AI-generated content has raised concerns about the potential threat to human jobs, particularly in the entertainment industry, with the example of an AI-generated actress, Tilly Norwood, sparking an uproar among actors, directors, and filmmakers 10s.
• The level of creativity and imagination in human-produced content, such as on the Hallmark Channel, can be extrapolated and interpolated by AI models, potentially threatening the jobs of working actors, directors, and screenwriters 2m6s.
• Merriam-Webster declared "slop" to be the word of the year, referring to low-quality content produced by AI, but it is argued that this definition is problematic as it implies that AI-generated content is easily identifiable and not competing with human content 4m42s.
• There are many examples of AI-generated content that are already being consumed by people, such as a YouTube Shorts channel dedicated to pythons and gorillas fighting, which has millions of views and is often indistinguishable from real content 6m15s.
• Tests have shown that around 20% of the content on YouTube Shorts and TikTok is already AI-generated, and real content from movies and TV shows is being filtered to make it look more AI-generated, making it harder to tell the difference between real and AI-generated content 8m10s.
• Many ads on YouTube and other websites are now AI-generated, including ones featuring celebrities like Oprah and Ben Carson, which can be misleading and difficult to distinguish from real endorsements 10m20s.
• AI-generated images, such as the "Tiananmen Square Tank Man selfie", can also be easily created and spread, with hundreds of copies of the image available online, highlighting the need to be aware of the potential for AI-generated content to be mistaken for real content 12m30s.

The Spread of AI-Generated Content and Misinformation

• The widespread use of AI tools has led to the creation of AI-generated content that can be found in various places, including peer-reviewed journals and Google search results, which can be misleading as it may be assumed to be real, 10s.
• A study by MIT Professor Sinan Aral showed that lies spread six times as fast on social media as the truth does, and this can be attributed to the fact that both the purveyors of lies and those who consume them contribute to their spread, 2m6s.
• The creation of convincing fakes requires a great deal of effort, talent, and resources, but with the advancement of technology, it has become easier to generate fake content, such as videos and audio, in a matter of seconds using systems like Sora and Grok, 4m42s.
• The use of generative AI has made it possible for one individual or entity to produce vast amounts of content, which can be used to spread misinformation, and this has been seen in the context of websites that are low quality and trying to attract AdSense viewers, 6m10s.
• The misinformation presented can be extremely subtle, and it is often difficult to determine whether the content is entirely AI-generated or if it was cleaned up or fact-checked by humans, 8m20s.
• A framework has been created to describe the stages of misinformation and fraud, which includes the creation of convincing fakes, and the spread of lies on social media, highlighting the need for awareness and caution when consuming online content, 10m30s.

Limitations and Risks of Generative AI

• Chat GPT, even with its most recent models, can provide confident but incorrect answers to simple counting questions, such as counting the number of J's in the last name "Ghost of Tsushima", and it is not recommended to use generative AI as a calculator due to its limitations in actual computing 10s.
• Generative AI models can create highly detailed diagrams, but they may contain misunderstandings or inaccuracies, and this can have serious consequences in fields like medicine, where precise information is crucial 2m6s.
• AI systems often get their facts from unexpected sources, including Wikipedia and Reddit, which can contain misinformation, and this can lead to the spread of false information 2m6s.
• There are instances of AI-generated content being presented as original, such as an article being translated into Spanish and published on a website, making it difficult to distinguish between genuine and generated content 4m42s.
• Venture capitalists have been known to use chatbots to gather information about companies, and these chatbots can provide misleading information, which can be attributed to the chatbot's reliance on AI-generated sources 6m15s.
• The introduction of chatbots with cited sources can create a false sense of trust, as the sources may be AI-generated, leading to the spread of misinformation, and this phenomenon is known as model collapse 8m30s.

AI in Cybercrime and Fraud

• Generative AI is considered the ultimate cybercriminal tool, as it has been used to clone voices and create deepfake representations, with the technology becoming increasingly sophisticated over time, as seen in the Arup case where an employee was tricked into transferring $25 million 42s.
• Cybercriminals have been automating their activities for the last 10 years, with organizations working together to create high levels of automation, and using techniques such as credential stuffing attacks to breach unrelated websites and accounts 2m6s.
• Credential stuffing attacks involve using stolen usernames and passwords to test them against login forms, with a typical success rate of 1 to 2%, allowing cybercriminals to take over thousands of accounts, and this is a common issue seen in every single industry 4m30s.
• Cybercriminals use specialized software, such as Century MBA, to plug into botnets and create large-scale credential stuffing attacks, which can be difficult to distinguish from organic user traffic 6m15s.
• The use of generative AI allows cybercriminals to automate their activities even further, making it a significant threat, and traditional protections such as CAPTCHA, which stands for Completely Automated Public Turing Test to tell computers and humans apart, have been tried but may not be effective against advanced bot-related activity 10m0s.
• The automation of cybercriminal activities is a highly commoditized and federated process, with organizations of cybercriminals working together to create high levels of automation, and this is a significant threat that is not well understood by the general public 8m40s.
• A study by Google found that humans had a 33% solve rate with typical synthetic distorted text CAPTCHAs, while machine learning-based OCR had a 99.8% solve rate, indicating that CAPTCHAs are no longer an effective mechanism against automated fraud 10s.
• Cybercriminal services specialize in helping other cybercriminals solve CAPTCHAs, offering group discounts and customer support, making it easier for them to bypass security measures 1m5s.
• Using CAPTCHAs today can actually introduce friction for real users while posing no barrier for cybercriminals, who can use generative AI systems to solve them easily 2m6s.

Automation and the Evolution of Cybercriminal Tactics

• The rise of automation for cybercriminals is exemplified by the IRS phone scams, which affected over 400,000 people in the United States, and were made possible by the use of call centers to convince victims to transfer money 3m15s.
• A study by Stanford found that generative AI has the most promising application in customer support, which can be exploited by cybercriminals to automate their operations and generate realistic audio and video to con people 4m30s.
• Generative AI can be used by cybercriminals to automate their "last mile problem" of generating realistic content to deceive victims, making it a powerful tool for scams 5m40s.
• Experts like Geoffrey Hinton and Warren Buffett have warned about the dangers of AI-enabled scams, with Buffett calling it the greatest growth industry he's ever seen, but one that he cannot invest in 7m10s.
• The democratization of AI has made it possible for cybercriminals to create effective AI systems at a low cost, contradicting the idea that AI is extremely expensive to create 8m50s.
• The key to fooling people effectively with AI does not require the greatest AI in the world, but rather a system that can generate believable content, making it a significant threat to security 10m20s.
• There are various examples of people being fooled at scale without sophisticated technology, and to address this, creating a secret password with family members can be a good exercise, but it may not be effective if fraudsters simulate a loved one's voice and claim to have forgotten the password 10s.
• Fraudsters do not need to be successful 100% of the time, as even a 2% success rate can be enough to create multi-billion dollar businesses, and common practices like phishing training or security training have drawbacks and may not prevent people from clicking on phishing links or engaging with social engineering attempts 2m6s.

Security Measures and Their Limitations

• Techniques like CAPTCHA, security training, and deep fake detection have limitations, with deep fake detection being difficult due to the widespread use of AI processing in images and videos, making it hard to definitively say if something is generated by a malicious deep fake model or not 4m30s.
• Multi-factor authentication and behavioral know your customer operations, which study the behavior of an account or device to flag anomalies, can be effective techniques, and the concept of zero trust security has risen in the last 10 years, where authentication gates are not trusted, and access is restricted even after passing the gate 8m40s.
• The idea of zero trust security is based on not giving full access to users after they pass an authentication gate, and instead, continuously monitoring and verifying their actions to prevent potential security threats 10m20s.

Zero Trust Security and Cyber Fusion

• The security industry has adopted a zero-trust mindset, which involves constantly monitoring account and device behavior to detect potential abuse, a concept that has always been present in the fraud industry, and this has led to increased collaboration between fraud teams and infosec teams 10s.
• Cyber fusion centers have been established to integrate efforts and utilize available data to identify patterns and anomalies, and spot potential abuse, which is a positive trend in the industry 2m6s.

AI's Impact on Different Areas of Cybersecurity

• There are three areas in organizations impacted by AI in different ways: infrastructure security, where AI enables cybercriminals to discover and exploit vulnerabilities at scale, business model and trust and safety, where account abuse and automation of user actions are concerns, and channels of communication, where AI can be used for social engineering 2m6s.
• AI can help cybercriminals analyze web applications for vulnerabilities, providing a more complete list of potential weaknesses to exploit, and this has led to warnings from organizations like Gartner to block AI-enabled browsers to mitigate this risk 2m6s.
• The use of AI-enabled browsers, such as those launched by OpenAI and Perplexity, has introduced new risks, including the potential for automation and account abuse, and organizations are advised to block these browsers to protect themselves 2m6s.
• Cybercriminals can use AI to socially engineer employees, customers, and executives, and to attack multiple targets simultaneously, making it essential for organizations to be aware of these risks and take steps to protect themselves 2m6s.

The Future of AI in Cybersecurity and Defense

• Automation has enabled cyber criminals to perform attacks that were previously blocked, and generative AI now allows them to realistically speak to victims on the phone, making it necessary to think differently about protecting communication channels against these types of attacks 10s.
• To combat automated fraud, good AI is needed to fight against bad AI, and scale is required to deal with the scale of cyber criminal attacks, with a study from MIT's Professor Tom Malone finding that humans and AI combined outperform humans alone 42s.
• The combination of humans and AI, also known as co-intelligence or human augmentation, is expected to lead to the development of products and services that help humans make better decisions, with generative AI serving as a fantastic brainstorming partner to refine thinking 2m6s.
• The future of AI is already here, but it's not evenly distributed, with examples of both dangerous and beneficial uses of AI existing, and it's essential to identify and extrapolate from these examples to figure out how to scale and use AI throughout organizations and society 2m6s.

Conclusion and Final Remarks

• It's crucial to monitor advancements in new technologies, including AI, to identify potential benefits and risks, and to use AI only when it makes life better, rather than using it for its own sake, with the goal of improving organizations and products quicker than competitors 2m6s.
• The person presenting is available to answer questions and can be approached in person after the presentation, and they have a social media policy of only adding people they have met in person, which applies to everyone present 10s.
• The presentation is now concluding, and the person is expressing gratitude to the audience, as indicated by the phrase "Thank you very much" 42s.
• Music is played after the conclusion of the presentation, signaling the end of the talk 42s.