Web Page Crawler Revealed - tracking down requests from Contabo GmbH

In my post about Scraper Junkyard results, I hypothesized a single, centrally-controlled web crawler that made requests from Contabo GmbH IP address ranges. After a closer look at the Apache combined format log files, I can confirm that at least one web crawler works out of Contabo IP address space. Contabo is a cloud provider, renting virtual private servers, and dedicated servers. This isn’t someone taking advantage of residential proxies. Above, a visualization of web traffic to my Scraper Junkyard, a large number of interlinked pragmatically generated URLs for the period 2026-08-01, when I made the scraper junkyard available, through 2026-09-30, when I lost patience with the experiment. The golden squares plot a subset of all the requests, the requests from Contabo IP addresses. Lines join actual data points to make it easier for human eyes to follow trends. The lines don’t mean anything. Scraper Junkyard The Scraper Junkyard web pages are all generated by a PHP program. Each web page has a “node number”, and the PHP program generates HTML with links to a parent node, left child node and right child node. If N is a page’s node number, the left child link is to a page with node number 2N+1, the right child link is to a page with node number 2N+2. The parent link is to node number (N-1)/2. Since the PHP program does integer arithmetic, the parent link node number is always an integer. The parent, left and right child hyperlinks of the generated web pages form a large binary tree, with the node number of each web page appearing in breadth first order. The Scraper Junkyard entry point, and root node of the binary tree is node number 0. It had the URL /node_0.html. The breadth-first web page node numbers become important further along, so here’s an elaboration: That’s the first four levels of the binary tree. The maximum generated node number was 1048578, so the tree had 20 full levels, and 4 nodes on level 21. I didn’t correctly calculate what node number to stop generating parent and child links. Internal Structure of Contabo Crawler Requests The gold and purple points and lines aren’t amenable to a whole lot of interpretation. There’s nothing more to think about. Luckily, the web pages have node numbers, arranged by hyperlink in a very specific manner. Looking at what node numbers Contabo IP addresses request shows a lot of structure. Here’s the node number of each request, against the Apache log file time stamp. That’s a whole lot of structure. I named …