Ask Your Question
3

As it parses, Python Scrapy is experiencing a decrease in speed.

asked 2023-05-21 09:45:19 +0000

lalupa gravatar image

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
3

answered 2023-05-21 09:59:02 +0000

pufferfish gravatar image

This could be due to several reasons such as:

  1. Too many requests: Scrapy may be making too many requests at a time which can slow down the parser. It's important to limit the number of requests and optimize their order.

  2. Large Responses: Large responses containing too much data can slow down the parser. It's best to limit the amount of data obtained in each response and only parse the required information.

  3. Overloaded CPU or RAM: If the CPU or RAM is overloaded, Scrapy can experience a decrease in speed. It's important to optimize system resources and reduce the load on the CPU and RAM.

  4. Slow Internet Connection: Slow internet can slow down the parser. It's best to ensure a stable and fast internet connection.

  5. Complex Parsing Logic: If the parsing logic is complex, it can take time to parse the data. It's important to optimize the parsing logic and reduce the time taken to parse the data.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account. This space is reserved only for answers. If you would like to engage in a discussion, please instead post a comment under the question or an answer that you would like to discuss

Add Answer


Question Tools

Stats

Asked: 2023-05-21 09:45:19 +0000

Seen: 7 times

Last updated: May 21 '23