Ask Your Question
2

As it parses, Python Scrapy is gradually decreasing in speed.

asked 2023-05-25 15:56:07 +0000

lakamha gravatar image

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
2

answered 2023-05-25 16:14:02 +0000

nofretete gravatar image

This could be due to several factors such as:

  1. Increased complexity of the website being scraped - if the website being scraped has a lot of pages or complex JavaScript, the parsing time may increase.

  2. Poorly optimized code - if the code is not optimized, it can slow down the parsing process.

  3. High traffic on the website - if there are a lot of people using the website at the same time, it can impact the speed of the parsing.

  4. Poor internet connection - slow internet speed can slow down the parsing process.

To address these issues, some possible solutions include optimizing the code, using proxies, limiting the number of concurrent requests or increasing the timeout values.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account. This space is reserved only for answers. If you would like to engage in a discussion, please instead post a comment under the question or an answer that you would like to discuss

Add Answer


Question Tools

Stats

Asked: 2023-05-25 15:56:07 +0000

Seen: 12 times

Last updated: May 25 '23