Revision history [back]

If scrapy is ineffective in gathering items, the following steps can be taken:

1. Check the website's robots.txt file to ensure that it is not blocking the crawler.
2. Verify that the xpath or css selectors are accurate and up to date.
3. Increase the download delay and/or the number of concurrent requests.
4. Use proxies to bypass IP blocking.
5. Consider using a different web scraping tool or library.
6. Contact the website owner to request permission to scrape their website.
7. Verify that the website is not using dynamic pages and that the scraper is configured to handle JavaScript and Ajax content.
8. If all else fails, manually extract the data by copying and pasting or using a web scraper extension in a web browser.