Lemmy newb here, not sure if this is right for this /c.

An article I found from someone who hosts their own website and micro-social network, and their experience with web-scraping robots who refuse to respect robots.txt, and how they deal with them.

  • drkt@lemmy.dbzer0.com
    link
    fedilink
    English
    arrow-up
    2
    ·
    3 days ago

    I wouldn’t even know where to begin, but I also don’t think that what I’m doing is anything special. These NVR IPs are hurling abuse at the whole internet. Anyone listening will have seen them, and anyone paying attention would’ve seen the pattern.

    The NVRs I get the most traffic from have been a known hacked IoT device for a decade and even has a github page explaining how to bypass their authentication and pull out arbitrary files like passwd.