The 2 A.M. Wake-Up Call

It’s usually about 2:00 AM on a Tuesday when the frantic Slack messages start pinging.

Customer support is reporting that "orders aren't showing up in the ERP." Marketing is screaming that new leads from Stripe aren't making it into HubSpot. You log in, bleary-eyed, staring at logs that seem perfectly fine on your end.

Then you check your payment processor's dashboard. You see 50 successful transactions in the last hour. You check your database. You only have 47 new orders.

Three orders vanished into the digital ether. Poof. Gone.

If your system relies 100% on webhooks to know when something happens in an external system, this isn't a "what if" scenario. It’s a mathematical certainty. It’s happening right now, probably in small enough numbers that you haven't noticed yet.

But when Black Friday hits, or you launch that big new feature, that tiny drip of lost data turns into a flood of angry customers and lost revenue.

After spending over a decade building integrations for high-volume e-commerce and SaaS platforms, I’ve learned one hard truth: Webhooks are a fantastic optimization, but they are a terrible source of truth.

If you want to sleep through the night, you need a safety net. You need a polling fallback.

The "Happy Path" Lie We Tell Ourselves

When we read API documentation for platforms like Shopify, Stripe, or Salesforce, they make webhooks sound flawless.

○ "Event occurs -> We push data to your URL -> Done."

It’s intoxicatingly simple. It feels modern. It’s real-time. Why would anyone want to go back to the dark ages of constantly asking an API, "Do you have anything new for me? How about now? How about now?"

So, we build the "Happy Path." We spin up an endpoint, maybe throw in a signature verification check to feel secure, process the payload right there in the request controller, return a 200 OK, and call it a day.

It works great in staging. It works great for the first month of production.

But the internet is messy. The internet is built on unreliable networks, fluctuating latencies, and servers that randomly decide to take a nap.

Where The "Push" Breaks Down

Here’s the reality of what happens between that external service and your server:

1. Your Server Blips: Your hosting provider has a momentary network hiccup. The webhook sender tries to connect, times out after 5 seconds, and gives up.
2. A Bad Deploy: You push bad code that throws a 500 error on the webhook endpoint for just 90 seconds before you roll it back. Any events sent during those 90 seconds are toast.

3. The Sender Has Issues: The third-party service’s outbound webhook queue gets backed up. They end up dropping older events to catch up.
4. Rate Limiting Your Own DB: Suddenly you get 1,000 webhook hits in one second. Your database connection pool locks up. Your app returns a 503 Service Unavailable. The sender treats this as a failure.

The critical problem is this: Most services have very mediocre retry logic for failed webhooks.

They might try three times over an hour. Some, like Stripe, are generous and try for days. Others try once, shrug, and mark it as failed.

If your server is down during that retry window, that data is gone forever. The external system thinks it told you about the order. You don't know it exists. This is the "split-brain" scenario that keeps lead engineers awake at night.

The Missing Piece: The Polling Fallback

So, if webhooks are unreliable, do we abandon them? No. They are still the best way to get low-latency updates.

But we change our mindset.

Webhooks are for speed. Polling is for consistency.

You need both.

A polling fallback is a background job a cron task, a scheduled worker that runs periodically (say, every 10 or 30 minutes). Its only job is to ask the external API: "Hey, give me everything that changed since the last time I checked."

It is your system's janitor. It sweeps the floor to find the crumbs that the webhook party dropped.

When you implement this, you will be shocked. I guarantee that within the first week, your logs will show the polling job finding orders or customer updates that your webhooks completely missed. It’s a humbling moment.

How to Build a "Smart" Polling Safety Net

Don't just hammer their API every minute requesting all orders. That’s how you get rate-limited and banned. You need to build an intelligent, cursor-based polling mechanism.

Here is the blueprint I’ve used to secure billions of dollars in transaction volume.

1. The Golden Rule: Idempotency

Before you write a line of polling code, your system must be idempotent.

Since you will now be receiving data via both webhooks and polling, you will receive the same event twice.

● The webhook might arrive successfully.
● Ten minutes later, your polling job might fetch the exact same order data.

If processing that order twice means charging the customer’s credit card twice or shipping two boxes, you’ve got a massive problem.

Your processing logic needs to check: " Have I already processed event ID evt_123xyz?" If yes, log it and discard it. If no, proceed.

2. The "Modified At" Cursor

You need to know where you left off the last time you polled. Don't rely on order IDs (they aren't always sequential). Rely on time.

Almost every decent API allows filtering resources by an updated_at, modified_after, or similar timestamp field.

The Workflow:

1. Store a State: In your database (Redis is great for this), store a timestamp called last_successful_poll_time. Initialize it to a safe time in the past (e.g., 24 hours ago).
2. The Job wake up: Your cron job runs. It reads the last_successful_poll_time.
3. Ask the API: You make a request to the external API asking for all records updated after that timestamp.
- Example request: GET /api/orders?updated_after=2023-10-27T10:00:00Z
4. Process the Batch: Iterate through the results. Pass each result to your (now idempotent) processor. It shouldn't matter if it's a new order or an update to an old one; your processor handles it.
5. Update the Cursor: This is vital. Find the latest updated_at timestamp in the batch you just fetched. Save that new timestamp back to your last_successful_poll_time storage.
6. Sleep: Wait for the next scheduled run.

3. The Overlap buffer

Don't trust clocks.

The clock on your server and the clock on the external API's server are never perfectly synced. There is always clock skew.

If you poll up to 10:00:00, and your next poll starts at 10:00:00, you might miss an event that happened at 10:00:00.001 on their server before the clocks aligned.

Always build in an overlap buffer. When you poll, ask for data from last_successful_poll_time minus 5 or 10 minutes.

Yes, this increases the number of duplicate events you’ll fetch. That’s why rule #1 (Idempotency) is non-negotiable. The safety is worth the extra processing cycles.

The Developer's Toolkit: Code and Config

Let’s look at how this changes your actual code structure. We'll use pseudo-Python/Flask for clarity, but the concepts apply to Node, Ruby, or Go.

The "Naive" Webhook Handler (Don't do this)

This is what most tutorials teach. It tries to do everything in the request pathway. It is brittle.

Python

# BAD EXAMPLE - DO NOT USE IN PRODUCTION

@app.route('/webhooks/stripe', methods=['POST'])
def handle_stripe_webhook():
    payload = request.json
    event = payload['data']['object']

    # DANGER ZONE: Performing heavy business logic right here.
    # If database is slow, this times out.
    # If external email service is down, this fails.
    try:
        order = database.save_order(event)
        email_service.send_confirmation(order.user_email) # What if this fails?
        return jsonify({'status': 'success'}), 200
    except Exception as e:
        # If we return 500, Stripe might retry.
        # But what if we already saved the order but failed the email?
        # Retry could cause duplicates if not idempotent.
        return jsonify({'error': str(e)}), 500

The "Pro" Approach: Async Webhooks + Polling

The robust architecture separates concerns.

1. The Webhook Receiver (Dumb and Fast)

Its only job is to take the payload, shove it into a message queue (like RabbitMQ, SQS, or Redis streams), return a 200 OK immediately, and terminate. It should execute in milliseconds.

Python

# BETTER EXAMPLE - Async Intake

@app.route('/webhooks/stripe', methods=['POST'])
def handle_stripe_webhook_fast():
    # 1. Verify signature (omitted for brevity, but essential)

    raw_payload = request.get_data(as_text=True)

    # 2. Shove into a queue immediately. Don't process it yet.
    # This is incredibly fast and rarely fails.
    redis_queue.enqueue('stripe_events', raw_payload)

    # 3. Tell Stripe "Got it, thanks."
    return jsonify({'status': 'received'}), 200

2. The Polling Worker (The Safety Net)

This runs separately on a schedule.

Python

# THE POLLING FALLBACK WORKER (Simplified Concept)
import time
import redis_client
import external_api_client

POLL_INTERVAL_SECONDS = 900 # 15 minutes
CURSOR_KEY = "last_stripe_poll_cursor"
OVERLAP_BUFFER_SECONDS = 300 # 5 minutes minutes

def run_polling_job():
    while True:
        print("Starting polling sweep...")

        # Get last cursor, default to 24 hours ago if first run
        last_poll_time = redis_client.get(CURSOR_KEY) or (time.time() - 86400)

        # Apply overlap buffer
        safe_start_time = float(last_poll_time) - OVERLAP_BUFFER_SECONDS

        # Format time for API (e.g., ISO 8601)
        api_time_string = format_to_iso(safe_start_time)

        # Fetch from external API
        # Pagination logic omitted for brevity, but you must handle it!
        events = external_api_client.get_events(updated_after=api_time_string)

        max_time_seen = last_poll_time

        for event in events:
            # Crucial: Send to the SAME idempotent processor the webhook uses
            process_event_idempotently(event)

            # Track the latest timestamp seen in this batch
            event_time = parse_iso_to_timestamp(event['updated_at'])
            if event_time > max_time_seen:
                max_time_seen = event_time

        # Update cursor for next time
        redis_client.set(CURSOR_KEY, max_time_seen)

        print(f"Sweep complete. Slept until next run.")
        time.sleep(POLL_INTERVAL_SECONDS)

The Business Value (How to Sell This to Your Boss)

If you are an engineer, you know this architecture is better. But your stakeholders might balk at the extra time required to build "the same integration twice" (once for webhooks, once for polling).

Here is how you frame it to non-technical leadership. Don't talk about "latency" and "TCP/IP stack failures." Talk about revenue risk.

The Pitch:

"Right now, our integration relies on a 'push' mechanism. It’s fast, but if there's a hiccup on the internet, the data is dropped. We wouldn't know we missed an order until a customer complains.

I want to build a safety mechanism. It takes a little longer now, but it acts as an automatic audit. It ensures that every dollar recorded in Stripe makes it into our ERP, guaranteed.

The cost of building this now is X hours. The cost of not building it is unknown lost revenue, manual reconciliation nightmares during peak season, and eroded customer trust. Which risk do you want to take?"

Final Verdict: Embrace the Belt and Suspenders

In the world of distributed systems, paranoia is a virtue.

Relying only on webhooks is like walking a tightrope without a net. Sure, you might make it across 99 times out of 100. But that one fall is going to hurt enormously.

Webhooks are great. Use them. Love them for their speed. But don't trust them with your life or your order database. Build the polling fallback. It’s unglamorous, backend plumbing work, but it’s the difference between a fragile system and an anti-fragile one that can survive the chaos of the real internet.