Bot Blocking By Big Box Retailers

Bot blocking prevents Australians from choice of search engines in the same way the Yellow Pages (decades ago)... if they had been prevented from knowing a business' name and phone number they would not be able to print it in their book (the same as Wallabyup not having a blue URL to click in search engine results). One of the difficulties new search engines face when entering the market is most big box retailers and ASX 200 sites block bots indiscriminately however they will allow bots like GoogleBot and BingBot to crawl their website. At the time of writing (early 2024)... Bing W, Target, The Good Guys, David Jones, Kmart, and many more are blocking bots, like WallabyupBot, from their website. Others, like Bunnings, allow just "store/location pages" and DIY tutorials however block ALL their products pages. Similarly Woolworths only allows pages like gift cards/insurance/etc. but blocks product pages. Jaycar used to allow all pages but from Dec? 2023 is now giving a 403 error (if their "anti-robot puzzle" is not solved by a human). The below tables are updated daily (last: 12/04/2025) and is sorted by the "ratio of pages indexed to unique backlinks". You basically want the ratio to be well over 1000% to indicate the retailer is not blocking bots. A backlink is a link from a 3rd party with more unique backlinks to products indicating a higher number of products on their website. See below table for actual index page counts. About the WallabyupBot bot; - it uses 2 different techniques to crawl a web page and differentiates between http httpS and the www subdomain. - it follows redirects (e.g. http to httpS). - if a websites page is listed in their robots.txt blocking rules or there is a meta tag blocking robots then WallabyupBot obeys (does not index). - any links found on a website with a no-follow tag will not be followed (note that 3rd party/off site no-follow outlinks are followed). Here is some more data from 12/04/2025... the individual stats of "Robots blocking ratio" and "Offline pages ratio" ratios should also each be well under 5%.

Robots Blocked Website Data And Stats

Retailer: rebelsport.com.au (Wal)
Ratio of pages indexed to backlinks: 0% (0 pages indexed by Wal / 255 unique links from other websites).
Robots blocking ratio: 60% (3 blocked pages / 5 total error pages).
Bad redirects ratio: 0% (0 errors / 5 total error pages).
Offline pages ratio: 40% (2 / 5 total error pages).


Retailer: thegoodguys.com.au (Wal)
thegoodguys.com.au's ISP "Akamai Technologies, Inc." has 1,973 domains with 6,965,046 pages in the Wallabyup index.

Ratio of pages indexed to backlinks: 0% (0 pages indexed by Wal / 487 unique links from other websites).
Robots blocking ratio: 55.6% (5 blocked pages / 9 total error pages).
Bad redirects ratio: 22.2% (2 errors / 9 total error pages).
Offline pages ratio: 22.2% (2 / 9 total error pages).


Retailer: target.com.au (Wal)
Ratio of pages indexed to backlinks: 0% (0 pages indexed by Wal / 801 unique links from other websites).
Robots blocking ratio: 75% (15 blocked pages / 20 total error pages).
Bad redirects ratio: 0% (0 errors / 20 total error pages).
Offline pages ratio: 0% (0 / 20 total error pages).


Retailer: bigw.com.au (Wal)
Ratio of pages indexed to backlinks: 0% (0 pages indexed by Wal / 1433 unique links from other websites).
Robots blocking ratio: 95.5% (42 blocked pages / 44 total error pages).
Bad redirects ratio: 2.3% (1 errors / 44 total error pages).
Offline pages ratio: 0% (0 / 44 total error pages).


Retailer: supercheapauto.com.au (Wal)
supercheapauto.com.au's ISP "Amazon Technologies Inc." has 21,880 domains with 14,112,227 pages in the Wallabyup index.

Ratio of pages indexed to backlinks: 0% (1 pages indexed by Wal / 281 unique links from other websites).
Robots blocking ratio: 40% (2 blocked pages / 5 total error pages).
Bad redirects ratio: 40% (2 errors / 5 total error pages).
Offline pages ratio: 20% (1 / 5 total error pages).


Retailer: davidjones.com (Wal)
davidjones.com's ISP "Incapsula Inc" has 699 domains with 1,644,354 pages in the Wallabyup index.

Ratio of pages indexed to backlinks: 0% (1 pages indexed by Wal / 639 unique links from other websites).
Robots blocking ratio: 46.2% (6 blocked pages / 13 total error pages).
Bad redirects ratio: 7.7% (1 errors / 13 total error pages).
Offline pages ratio: 0% (0 / 13 total error pages).


Retailer: coles.com.au (Wal)
coles.com.au's ISP "Incapsula Inc" has 699 domains with 1,644,354 pages in the Wallabyup index.

Ratio of pages indexed to backlinks: 1% (24 pages indexed by Wal / 2356 unique links from other websites).
Robots blocking ratio: 47.3% (61 blocked pages / 129 total error pages).
Bad redirects ratio: 15.5% (20 errors / 129 total error pages).
Offline pages ratio: 1.6% (2 / 129 total error pages).


Retailer: woolworths.com.au (Wal)
woolworths.com.au's ISP "Akamai Technologies, Inc." has 1,973 domains with 6,965,046 pages in the Wallabyup index.

Ratio of pages indexed to backlinks: 1% (42 pages indexed by Wal / 3099 unique links from other websites).
Robots blocking ratio: 53.3% (65 blocked pages / 122 total error pages).
Bad redirects ratio: 2.5% (3 errors / 122 total error pages).
Offline pages ratio: 0.8% (1 / 122 total error pages).


Retailer: kmart.com.au (Wal)
kmart.com.au's ISP "Incapsula Inc., ABUSE INCAPSULAAP, INCAPSULA INC - Network administrator" has 16 domains with 3,318 pages in the Wallabyup index.

Ratio of pages indexed to backlinks: 2% (21 pages indexed by Wal / 1400 unique links from other websites).
Robots blocking ratio: 66.7% (54 blocked pages / 81 total error pages).
Bad redirects ratio: 17.3% (14 errors / 81 total error pages).
Offline pages ratio: 8.6% (7 / 81 total error pages).


Retailer: harveynorman.com.au (Wal)
harveynorman.com.au's ISP "Incapsula Inc" has 699 domains with 1,644,354 pages in the Wallabyup index.

Ratio of pages indexed to backlinks: 16% (123 pages indexed by Wal / 774 unique links from other websites).
Robots blocking ratio: 84.2% (32 blocked pages / 38 total error pages).
Bad redirects ratio: 0% (0 errors / 38 total error pages).
Offline pages ratio: 2.6% (1 / 38 total error pages).


Retailer: jbhifi.com.au (Wal)
jbhifi.com.au's ISP "Cloudflare, Inc." has 46,892 domains with 17,299,001 pages in the Wallabyup index.

Ratio of pages indexed to backlinks: 16% (217 pages indexed by Wal / 1362 unique links from other websites).
Robots blocking ratio: 4.9% (2 blocked pages / 41 total error pages).
Bad redirects ratio: 12.2% (5 errors / 41 total error pages).
Offline pages ratio: 26.8% (11 / 41 total error pages).


Retailer: kathmandu.com.au (Wal)
kathmandu.com.au's ISP "Cloudflare, Inc." has 46,892 domains with 17,299,001 pages in the Wallabyup index.

Ratio of pages indexed to backlinks: 48% (85 pages indexed by Wal / 177 unique links from other websites).
Robots blocking ratio: 15.4% (2 blocked pages / 13 total error pages).
Bad redirects ratio: 0% (0 errors / 13 total error pages).
Offline pages ratio: 7.7% (1 / 13 total error pages).


Retailer: officeworks.com.au (Wal)
officeworks.com.au's ISP "Amazon Technologies Inc." has 21,880 domains with 14,112,227 pages in the Wallabyup index.

Ratio of pages indexed to backlinks: 70% (647 pages indexed by Wal / 928 unique links from other websites).
Robots blocking ratio: 73.1% (19 blocked pages / 26 total error pages).
Bad redirects ratio: 15.4% (4 errors / 26 total error pages).
Offline pages ratio: 7.7% (2 / 26 total error pages).


Retailer: myer.com.au (Wal)
myer.com.au's ISP "Amazon.com, Inc." has 6,999 domains with 2,774,964 pages in the Wallabyup index.

Ratio of pages indexed to backlinks: 70% (669 pages indexed by Wal / 961 unique links from other websites).
Robots blocking ratio: 54.3% (25 blocked pages / 46 total error pages).
Bad redirects ratio: 15.2% (7 errors / 46 total error pages).
Offline pages ratio: 0% (0 / 46 total error pages).


Retailer: bcf.com.au (Wal)
bcf.com.au's ISP "Amazon Technologies Inc." has 21,880 domains with 14,112,227 pages in the Wallabyup index.

Ratio of pages indexed to backlinks: 112% (293 pages indexed by Wal / 262 unique links from other websites).
Robots blocking ratio: 83.5% (162 blocked pages / 194 total error pages).
Bad redirects ratio: 0.5% (1 errors / 194 total error pages).
Offline pages ratio: 0.5% (1 / 194 total error pages).


Retailer: chemistwarehouse.com.au (Wal)
chemistwarehouse.com.au's ISP "Cloudflare, Inc." has 46,892 domains with 17,299,001 pages in the Wallabyup index.

Ratio of pages indexed to backlinks: 480% (7102 pages indexed by Wal / 1480 unique links from other websites).
Robots blocking ratio: 10.9% (62 blocked pages / 569 total error pages).
Bad redirects ratio: 0.4% (2 errors / 569 total error pages).
Offline pages ratio: 0.7% (4 / 569 total error pages).


Retailer: jaycar.com.au (Wal)
jaycar.com.au's ISP "Google LLC" has 38,034 domains with 2,433,563 pages in the Wallabyup index.

Ratio of pages indexed to backlinks: 483% (1058 pages indexed by Wal / 219 unique links from other websites).
Robots blocking ratio: 0% (0 blocked pages / 14 total error pages).
Bad redirects ratio: 7.1% (1 errors / 14 total error pages).
Offline pages ratio: 92.9% (13 / 14 total error pages).


Retailer: bunnings.com.au (Wal)
bunnings.com.au's ISP "Cloudflare, Inc." has 46,892 domains with 17,299,001 pages in the Wallabyup index.

Ratio of pages indexed to backlinks: 1553% (47960 pages indexed by Wal / 3088 unique links from other websites).
Robots blocking ratio: 21.9% (15457 blocked pages / 70741 total error pages).
Bad redirects ratio: 2.2% (1577 errors / 70741 total error pages).
Offline pages ratio: 6% (4267 / 70741 total error pages).


Retailer: clarkrubber.com.au (Wal)
Ratio of pages indexed to backlinks: 3421% (1779 pages indexed by Wal / 52 unique links from other websites).
Robots blocking ratio: 31.9% (84 blocked pages / 263 total error pages).
Bad redirects ratio: 1.1% (3 errors / 263 total error pages).
Offline pages ratio: 5.7% (15 / 263 total error pages).


Retailer: machineryhouse.com.au (Wal)
machineryhouse.com.au's ISP "Cloudflare, Inc." has 46,892 domains with 17,299,001 pages in the Wallabyup index.

Ratio of pages indexed to backlinks: 4657% (2282 pages indexed by Wal / 49 unique links from other websites).
Robots blocking ratio: 0% (0 blocked pages / 60 total error pages).
Bad redirects ratio: 0% (0 errors / 60 total error pages).
Offline pages ratio: 100% (60 / 60 total error pages).


Retailer: totaltools.com.au (Wal)
totaltools.com.au's ISP "Fastly, Inc." has 3,526 domains with 2,301,717 pages in the Wallabyup index.

Ratio of pages indexed to backlinks: 15928% (25167 pages indexed by Wal / 158 unique links from other websites).
Robots blocking ratio: 5% (77 blocked pages / 1551 total error pages).
Bad redirects ratio: 1.8% (28 errors / 1551 total error pages).
Offline pages ratio: 10.9% (169 / 1551 total error pages).


Retailer: scorptec.com.au (Wal)
scorptec.com.au's ISP "Cloudflare, Inc." has 46,892 domains with 17,299,001 pages in the Wallabyup index.

Ratio of pages indexed to backlinks: 21416% (21844 pages indexed by Wal / 102 unique links from other websites).
Robots blocking ratio: 3.6% (90 blocked pages / 2490 total error pages).
Bad redirects ratio: 0% (1 errors / 2490 total error pages).
Offline pages ratio: 19% (474 / 2490 total error pages).