How to Keep a Scraper Running 24/7

A web scraper script is often one of the first things a developer builds. It's a tangible way to interact with the web, gather data, and create value. Writing a script that runs once is straightforward. Making that same script run continuously, reliably, and without human intervention is a completely different class of problem. It's the transition from writing a program to engineering a service.

Achieving 24/7 uptime for a scraper isn't about a clever while True loop or a perfectly crafted try/except block. Those are necessary but insufficient. True continuous operation is a systems design challenge. It requires a stack of technologies and practices designed to handle failure automatically. This involves process management, scheduling, health monitoring, and often, a distributed architecture that is resilient by design. Understanding these layers is essential for any engineer or technical leader tasked with building a data pipeline that can't afford to go down.

From Fragile Process to Managed Service

A script running in a terminal is a fragile entity. It exists as a single process. If you close the terminal, the process receives a hangup signal (SIGHUP)^[1] and typically terminates. If the machine reboots, the process is gone. If the script encounters an unhandled exception or a memory leak that invokes the kernel's OOM Killer, it stops. The fundamental goal of 24/7 operation is to move from this fragile, manually-started process to an automatically managed service.

A service, in this context, is a program that is managed by a higher-level supervisor. The supervisor's job is simple but critical: ensure the application process is running. If the process dies for any reason, the supervisor's responsibility is to detect the failure and restart it. This principle is the foundation of high-availability systems and the first major step beyond running a script by hand.

Level 1: The Single-Machine Supervisor

The most direct way to create a managed service is to use a process supervisor on a single server, such as a cloud virtual machine. Modern Linux distributions come with powerful, built-in supervisors as part of their init system. The dominant init system today is systemd^[2], which replaced older systems like System V init^[3] and Upstart.

How systemd Works

systemd manages user and system services on a Linux machine. You define your scraper as a service by writing a simple configuration file called a unit file. This file tells systemd what command to execute, under which user to run it, and what to do if it stops. systemd's core process, which runs with Process ID 1^[4], monitors and manages all services defined on the system.

When you enable and start a service, systemd forks a new process to run your script. It then watches that process. If the process exits, systemd checks the unit file's Restart policy. If configured to restart (e.g., Restart=always), systemd will immediately start a new instance of your script, effectively healing the failure automatically. It also provides standardized tools for viewing logs (journalctl) and checking status (systemctl status).

Example: A Python Scraper as a systemd Service

Imagine a simple scraper that occasionally fails. It might look something like this:

# scraper.py
import requests
import time
import random
import logging

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

def scrape_site():
 """Scrapes a site, with a chance of failure."""
 logging.info("Starting a new scrape cycle.")
 try:
 # Simulate a random failure
 if random.random() < 0.1:
 raise ConnectionError("Failed to connect to proxy")

 response = requests.get("http://httpbin.org/get", timeout=10)
 response.raise_for_status() # Raise an exception for bad status codes
 logging.info(f"Successfully scraped. Status: {response.status_code}")

 except requests.exceptions.RequestException as e:
 logging.error(f"Scrape failed: {e}")
 # In a real app, this error would cause the script to exit if not caught in the main loop.
 # For this example, we'll let systemd handle a hard exit.
 exit(1)

if __name__ == "__main__":
 while True:
 scrape_site()
 time.sleep(30) # Wait 30 seconds between scrapes

To manage this with systemd, you create a service file at /etc/systemd/system/scraper.service:

[Unit]
Description=My 24/7 Python Scraper
After=network.target

[Service]
User=appuser
Group=appgroup
WorkingDirectory=/opt/scraper
ExecStart=/usr/bin/python3 /opt/scraper/scraper.py
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Key directives here:

• Description: A human-readable name.
• After=network.target: Ensures the network is up before starting.
• User/Group: Runs the process with lower privileges for security.
• ExecStart: The command to run your scraper.
• Restart=always: The magic ingredient. It tells systemd to restart the service whether it exits cleanly or with an error.
• RestartSec=10: Waits 10 seconds before attempting a restart to prevent rapid-fire failures.
• WantedBy=multi-user.target: Allows the service to be enabled to start on boot.

With this file in place, you can control your scraper with standard system commands:
sudo systemctl enable scraper.service (start on boot)
sudo systemctl start scraper.service (start now)
sudo systemctl status scraper.service (check status and see recent logs)

Tradeoffs and Decision Lens

A systemd service is a massive improvement over a manual process. It's simple, reliable for single-server needs, and uses tools already on every modern Linux server.

• When to use it: Ideal for small to medium-scale scrapers, internal tools, or projects where the operational complexity and cost of a distributed system are not justified. If your entire application lives on one or two VMs, systemd is a pragmatic and powerful choice.
• Failure Modes: Its primary weakness is being a single point of failure. If the entire virtual machine goes down, your scraper is offline. It also doesn't inherently solve application-level hangs where the process is alive but unproductive. It only reacts to process termination.

Level 2: Containerization and Orchestration

For business-critical scraping or operations at a larger scale, the industry standard is to use containers and an orchestrator. This approach treats infrastructure as code and is designed for high availability and scalability from the ground up.

How It Works: Docker and Kubernetes

First, the application is packaged into a container using a tool like Docker^[5]. A container bundles the scraper code with all its dependencies (libraries, interpreters, system tools) into a single, immutable artifact. This solves environment inconsistencies and makes the application portable. This standardization is governed by the Open Container Initiative (OCI)^[6].

Next, a container orchestrator like Kubernetes^[7] is used to run and manage these containers. Kubernetes is an open-source platform originally designed by Google for automating the deployment, scaling, and management of containerized applications.

For a 24/7 scraper, you use a Kubernetes object called a Deployment. A Deployment's job is to maintain a set of identical running containers, called Pods. You declare, "I want 1 instance of my scraper container running at all times." The Kubernetes control plane then works continuously to meet that declaration. If a node (machine) fails or a container crashes, the Deployment's controller notices the discrepancy and automatically schedules a new Pod on a healthy node to replace the lost one. This provides a much higher level of availability than a single-server systemd service.

For scrapers that run on a schedule (e.g., "scrape this site every hour") rather than continuously, Kubernetes provides a CronJob object. It runs a job on a time-based schedule defined by cron syntax^[8]. This is highly efficient, as resources are only consumed when the scraper is actively running.

Example: A Scraper as a Kubernetes CronJob

First, create a Dockerfile to containerize the Python script:

# Dockerfile
FROM python:3.9-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY scraper.py .

CMD ["python", "scraper.py"]

Next, define a Kubernetes CronJob in a YAML file:

# scraper-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
 name: hourly-scraper
spec:
 schedule: "0 * * * *" # Run at the start of every hour
 jobTemplate:
 spec:
 template:
 spec:
 containers:
 - name: scraper
 image: your-repo/scraper:v1.0.0 # Your container image
 restartPolicy: OnFailure
 concurrencyPolicy: Forbid # Don't run two jobs at once if one overruns

Key fields explained:

• schedule: The cron schedule. "0 * * * *" means on the zeroth minute of every hour.
• jobTemplate: A blueprint for the Job that will be created on schedule.
• image: The container image you built and pushed to a registry.
• restartPolicy: OnFailure: If the container inside the Pod exits with an error, the Job will try to restart it. This handles script-level failures.
• concurrencyPolicy: Forbid: Prevents a new job from starting if the previous one is still running.

To apply this, you use the Kubernetes command-line tool, kubectl:
kubectl apply -f scraper-cronjob.yaml

Kubernetes now ensures this job runs every hour, automatically handling node failures and script crashes.

Tradeoffs and Decision Lens

Kubernetes offers immense power, but it comes with significant complexity and cost.

• When to use it: For critical data pipelines where downtime has a direct business cost. When you need to scale your scraping efforts horizontally by running many scrapers in parallel. When your organization is already committed to a container-based infrastructure.
• Costs and Complexity: Running a Kubernetes cluster has a cost, both in cloud provider fees for the control plane and nodes, and in the engineering effort required to maintain it. For simpler tasks, it can be a premature optimization. Managed container platforms like AWS Fargate^[9] or Google Cloud Run^[10] can offer a middle ground, providing the benefits of containerization without the need to manage the underlying cluster.

Beyond Process Uptime: Ensuring Scraper Health

Keeping the process running is only half the battle. A scraper can be "running" but completely unproductive. It could be stuck in a network request, caught in an infinite loop, or consistently failing to parse pages due to a site layout change. A true 24/7 system must monitor for these application-level failures.

Health Checks and Probes

Modern supervisors and orchestrators support health checks. Instead of just checking if the process ID exists, they actively probe the application to see if it's healthy. Kubernetes formalizes this with liveness and readiness probes^[11].

• Liveness Probe: Kubernetes periodically checks a specific endpoint in your container (e.g., an HTTP endpoint at /healthz). If the probe fails a configured number of times, Kubernetes concludes the application is deadlocked and kills the container. The Deployment then replaces it, forcing a restart.
• Readiness Probe: This tells Kubernetes when your application is ready to start accepting traffic. While less critical for a standalone scraper, it's essential for services that need a startup period before they are fully functional.

Implementing a health check can be as simple as adding a small web server (like Flask^[12] in Python) to your scraper that returns a 200 OK response from a /healthz endpoint.

Monitoring, Logging, and Alerting

To run a service 24/7, you need visibility into its behavior. You cannot rely on manually checking logs on a server.

1. Centralized Logging: Configure your scraper to output structured logs (e.g., in JSON format). Ship these logs to a centralized platform like the ELK Stack^[13] (Elasticsearch, Logstash, Kibana), Datadog, or AWS CloudWatch Logs. This allows you to search, analyze, and dashboard log data from all your scrapers in one place.

2. Application Metrics: Instrument your code to track key performance indicators. For a scraper, this isn't just CPU and memory. It's business-level metrics like items_scraped_per_minute, pages_failed_rate, or captcha_detections. The standard tool for this is Prometheus^[14], an open-source monitoring system. Your application exposes these metrics on an HTTP endpoint, and Prometheus scrapes it periodically. You can then use a tool like Grafana^[15] to build dashboards and visualize trends.

3. Alerting: Proactive alerting is the final piece. Set up rules based on your metrics and logs. If items_scraped_per_minute drops to zero for more than 10 minutes, or if the HTTP 5xx error rate exceeds 5%, an alert should be automatically sent to your team via a tool like PagerDuty or Slack. This turns an unknown failure into a notified, actionable event.

The Distributed Scraping Architecture

As scale increases, a single monolithic scraper process becomes a bottleneck. The standard solution is to move to a distributed "queue and worker" architecture. This design is inherently more resilient and scalable.

• URL Queue: Instead of keeping a list of URLs to scrape in memory, they are pushed into a durable, centralized message queue like Redis^[16], RabbitMQ^[17], or a cloud service like AWS SQS^[18]. This queue acts as the system's source of truth for work to be done.
• Stateless Workers: You run multiple identical, stateless worker processes. Each worker's job is simple: pull a URL from the queue, scrape it, put the result in a database, and repeat. These workers can be systemd services or, more commonly, Pods in a Kubernetes Deployment.
• Scheduler/Feeder: A separate process is responsible for discovering URLs and populating the queue. This might be a Kubernetes CronJob that runs daily to find new articles on a homepage.

This architecture provides 24/7 operation because the components are decoupled and disposable. If a worker container crashes, the orchestrator replaces it. The new worker just pulls the next URL from the queue. The state of the system (the work to be done) is safely stored in the persistent queue, not in the memory of any single fragile process.

Common Misconceptions

Misconception: "A while True loop with a try/except block is enough for 24/7 operation."
Reality: This pattern only handles exceptions within the script's logic. It offers no protection against process termination from outside forces like a machine reboot, a deployment, or the OOM Killer. It provides no external visibility or automatic restarts.

Misconception: "Running 24/7 means a single process must never stop."
Reality: True 24/7 reliability embraces failure, it doesn't prevent it. The goal is to ensure the service is always available, which is achieved by having a system that can automatically replace failed components. This is the famous "pets vs. cattle" analogy: you don't nurse a single, precious server (a pet); you manage a herd of identical, disposable units (cattle).

Misconception: "The main challenge is avoiding getting blocked."
Reality: Avoiding blocks with proxies, user agents, and rate limiting is about scraper effectiveness. Ensuring the service runs 24/7 is about availability. They are distinct, though related, engineering challenges. A highly effective scraper that crashes every hour is not a 24/7 service.

The Decision Framework

As a technical leader, choosing the right approach depends on balancing reliability requirements with cost and complexity.

• Criticality: Is this data for a hobby project or a core business function? The higher the business impact of downtime, the stronger the case for a containerized, orchestrated solution.
• Scale: For a few hundred pages a day, a systemd service on a single VM is perfectly adequate. For millions of pages a day, a distributed, queue-based architecture is necessary to parallelize the work.
• Team Expertise: Does your team have experience with Docker and Kubernetes? If not, the learning curve is steep. A managed container platform or a simpler VM-based approach may deliver value faster and with less operational risk.
• Cost: A single VM might cost $10 per month. A managed Kubernetes cluster can cost hundreds or thousands, before even considering the cost of worker nodes. Always weigh the infrastructure and engineering costs against the value of the data and the cost of downtime.

Ultimately, building a scraper that runs 24/7 is an exercise in reliability engineering. It begins with shifting your mindset from writing a script to building a resilient, observable, and automated system.