The ServiceNow REST pagination gotcha that silently drops records

ServiceNow’s REST API uses offset-based pagination. When you paginate through a live table—one where records are being created or deleted—the offset becomes stale between calls. Records silently get dropped.

Why it happens

The offset is positional within the current result set at the exact moment you query. If a new incident is created and sorts before your current page, all subsequent offsets shift down—you skip records.

Example: You’re processing incidents with offset=0, limit=100, then offset=100, limit=100. If a new incident arrives and sorts into position 50, the second page now contains records 151–250 instead of 100–200. Record 100 is never seen.

The fix: use sys_id as a cursor

Instead of offset, anchor pagination to sys_id. Since sys_id is immutable, new records don’t invalidate your position—they sort after your cursor.

# DON'T: offset drifts
offset = 0
while True:
    records = api.query(offset=offset, limit=100)
    process(records)
    offset += 100  # Invalid if records were added

# DO: stable cursor
last_sys_id = ""
while True:
    query = f"sys_id>{last_sys_id}^ORDERBYsys_id" if last_sys_id else "ORDERBYsys_id"
    records = api.query(query, limit=100)
    process(records)
    if records:
        last_sys_id = records[-1]["sys_id"]  # Advance cursor

See the full before-and-after in github.com/waghmaredb/vexpose-labs.