Revision history [back]

There are various ways to store the output of a Scrapy spider outside of a Docker container. Here are a few options:

1. Mount a local directory as a volume in the Docker container and instruct Scrapy to save its output to that directory. This will allow you to access the files from outside the container. For example:

docker run -v /path/on/host:/output scrapy-image scrapy crawl spider -o /output/output.json

1. Use a cloud storage service, such as Amazon S3, to store the output. You can configure Scrapy to save its output directly to the cloud storage service. For example:

FEED_URI = 's3://bucket-name/spider-output/%(name)s-%(time)s.json'

1. Use a container orchestration platform, such as Kubernetes or Docker Swarm, to manage your Docker containers and store the output. These platforms can provide persistent storage solutions that allow you to store the output even if the container is destroyed or recreated.

2. Store the output in a database, such as MongoDB, and configure Scrapy to save its output to the database. This will allow you to access the data from outside the container and easily query it.

Overall, the best approach will depend on your specific use case and requirements.