Amazon Web Services (AWS) has recently launched an investigation into Perplexity AI to determine whether the AI search startup is violating AWS rules by scraping websites that have tried to prevent it. The investigation stems from concerns raised by the use of content from scraped websites that had explicitly forbidden access through the Robots Exclusion Protocol.
Scrutiny of Perplexity’s practices intensified following a report from Forbes accusing the startup of stealing one of its articles. WIRED also conducted investigations that confirmed the allegations, uncovering evidence of scraping abuse and plagiarism by systems associated with Perplexity’s AI-powered search chatbot.
While the Robots Exclusion Protocol is not a legally binding standard, companies are generally expected to respect it. AWS customers, including Perplexity AI, are required to adhere to the robots.txt standard when crawling websites. The AWS spokesperson emphasized that customers must not engage in any illegal activities and are responsible for compliance with the company’s terms and all relevant laws.
Despite efforts by engineers at Condé Nast to block Perplexity’s crawler using a robots.txt file, it was discovered that the startup had gained access to Condé Nast properties through an unpublished IP address. This IP address, 44.221.181.252, was observed making numerous visits to Condé Nast websites, indicating widespread scraping activities.
The IP address associated with Perplexity was traced to an Elastic Compute Cloud (EC2) instance hosted on AWS. This discovery prompted AWS to launch its own investigation into whether the use of its infrastructure to scrape websites that prohibit such activity violates the company’s terms of service.
In response to the investigations, Perplexity CEO Aravind Srinivas initially downplayed the allegations, suggesting a misunderstanding of the company’s operations. However, when pressed further, he attributed the scraping activities to a third-party company that provides web crawling and indexing services, refusing to disclose the company’s identity due to a nondisclosure agreement.
The controversy surrounding Perplexity AI highlights the complexities and ethical considerations involved in web scraping and data collection. As companies continue to leverage AI technologies for search and indexing purposes, it is crucial for them to operate within legal and ethical boundaries to protect the rights and interests of content creators and website owners.
Leave a Reply