During a content-heavy client news-site migration, I wrote my own importer to move an XML export off the old site. The export was big: hundreds of posts plus 465 attachments that had to be downloaded and registered into the media library. My first version of the importer was naive: one request, one loop, process everything until done. I clicked the import button, leaned back, and watched the spinner.
About a hundred seconds later, the spinner stopped. Not with a success message, and not with a PHP error either. What appeared was a Cloudflare-branded page: error 524 A Timeout Occurred. And the worst part was not the timeout itself. The worst part was that the import had died mid-run in an unknown state: some posts were in, some were not, and I had no idea where the line was.
Why the import died
My first instinct pointed the wrong way: I raised max_execution_time in PHP and threw set_time_limit(0) into the script. It changed nothing, and it never could have. PHP was not the killer. The proxy in front of it was.
Cloudflare has a hard cap on how long it will wait for a response from your origin: 100 seconds. Past that, Cloudflare stops waiting and serves the 524 page to the browser. That number is not configurable on regular plans, and here is the commonly misunderstood point: a 524 means your origin was too slow for Cloudflare, not that Cloudflare is down. My server was alive, the PHP process may well have kept running blind in the background, but the connection to the visitor was already severed. Importing 465 attachments in a single request was never going to finish under 100 seconds.
The conclusion is more general than this one incident: any long-running web job behind a proxy will get killed, sooner or later. Not maybe — will. So the design had to change, not the timeout settings.
The fix: a resumable batched state machine
I rebuilt the importer as a state machine: each request processes one small batch, persists its progress, then triggers the next request. Batches of 25 items turned out to be a comfortable size, finishing well under 100 seconds even while downloading attachments.
define( 'IMPORT_BATCH_SIZE', 25 );
function import_run_batch() {
$state = get_option( 'import_state', array( 'offset' => 0 ) );
$items = import_read_items( $state['offset'], IMPORT_BATCH_SIZE );
foreach ( $items as $item ) {
// Idempotent: skip anything a previous (crashed) run already imported.
if ( import_find_existing( $item['source_id'] ) ) {
continue;
}
$post_id = import_create_post( $item );
update_post_meta( $post_id, '_import_source_id', $item['source_id'] );
}
$state['offset'] += count( $items );
update_option( 'import_state', $state, false );
// True while there is still work left.
return count( $items ) === IMPORT_BATCH_SIZE;
}Two decisions in there saved me. First, progress is persisted to an option after every batch, so if anything dies, I know exactly where it stopped. Second, every item is checked against a _import_source_id meta before it gets created. That is what makes each batch idempotent: crash whenever, re-run whenever, and there will never be a duplicate post. It also cleaned up the "some in, some not" mess from the first run for free, because the next run simply skipped everything that already existed.
So how does the next request get triggered? I did not want to depend on JavaScript, and WP-Cron on that migration site was unreliable. The oldest trick turned out to be the most robust one: a meta http-equiv="refresh" redirect back to the importer URL.
if ( import_run_batch() ) {
$next = admin_url( 'admin.php?page=my-importer&resume=1' );
echo '<meta http-equiv="refresh" content="1;url=' . esc_url( $next ) . '">';
echo '<p>Batch imported, continuing...</p>';
exit;
}
delete_option( 'import_state' ); // done
echo '<p>Import finished.</p>';Every request finishes in seconds, far below Cloudflare's radar, and the browser itself asks for the next batch. No JS, no cron, no AJAX. The page just keeps refreshing until all 465 attachments are in.
The bonus trap: control bytes in the source XML
In the middle of that now-tidy run, the XML parser suddenly threw on certain items. It turned out the export from the old site contained raw control characters — bytes that are invalid in XML — buried inside some of the content. A spec-compliant parser rightfully refuses them. The fix: sanitize the bytes before parsing, stripping every control character except tab, newline, and carriage return.
// Strip control bytes the XML parser chokes on (keep tab, LF, CR).
$xml = preg_replace( '/[^\x09\x0A\x0D\x20-\x{10FFFF}]/u', '', $xml );Never assume third-party XML is clean. One stray byte is enough to take down the entire run.
The takeaway
My checklist now for any long web-based job: first, assume the proxy will kill you, and design for resumability from day one. Second, small batches, persist state after each one, and make every batch idempotent with a source-id check. Third, sanitize third-party XML before it reaches the parser. And fourth, when you see a 524, stop blaming Cloudflare: it is your origin that took too long to answer. Since the importer became a state machine, 465 stopped being a threat — it is just 19 page refreshes.