• RelativeArea1@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    82
    arrow-down
    1
    ·
    edit-2
    15 days ago

    this is some fucking stupid situation, we somewhat got a faster internet and these bots messing each other are hogging the bandwidth.

    • dual_sport_dork 🐧🗡️@lemmy.world
      link
      fedilink
      English
      arrow-up
      9
      arrow-down
      1
      ·
      edit-2
      15 days ago

      Especially since the solution I cooked up for my site works just fine and took a lot less work. This is simply to identify the incoming requests from these damn bots – which is not difficult, since they ignore all directives and sanity and try to slam your site with like 200+ requests per second, that makes 'em easy to spot – and simply IP ban them. This is considerably simpler, and doesn’t require an entire nuclear plant powered AI to combat the opposition’s nuclear plant powered AI.

      In fact, anybody who doesn’t exhibit a sane crawl rate gets blocked from my site automatically. For a while, most of them were coming from Russian IP address zones for some reason. These days Amazon is the worst offender, I guess their Rufus AI or whatever the fuck it is tries to pester other retail sites to “learn” about products rather than sticking to its own domain.

      Fuck 'em. Route those motherfuckers right to /dev/null.

    • IninewCrow@lemmy.ca
      link
      fedilink
      English
      arrow-up
      5
      arrow-down
      2
      ·
      15 days ago

      It’s what I’ve been saying about technology for the past decade or two … we’ve hit an upper limit to our technological development … that limit is on individual human greed where small groups of people or massively wealthy people hinder or delay any further development because they’re always trying to find ways to make money off it, prevent others from making money off it, monopolize an area or section of society … capitalism is literally our world’s bottleneck and it’s being choked off by an oddly shaped gold bar at this point.

    • Dr. Moose@lemmy.world
      link
      fedilink
      English
      arrow-up
      1
      ·
      15 days ago

      Lol website traffic accounts for like 1% of bandwidth budget. 1 netflix movie is like 20k web pages.

  • umbraroze@lemmy.world
    link
    fedilink
    English
    arrow-up
    33
    ·
    14 days ago

    I have no idea why the makers of LLM crawlers think it’s a good idea to ignore bot rules. The rules are there for a reason and the reasons are often more complex than “well, we just don’t want you to do that”. They’re usually more like “why would you even do that?”

    Ultimately you have to trust what the site owners say. The reason why, say, your favourite search engine returns the relevant Wikipedia pages and not bazillion random old page revisions from ages ago is that Wikipedia said “please crawl the most recent versions using canonical page names, and do not follow the links to the technical pages (including history)”. Again: Why would anyone index those?

    • Phoenixz@lemmy.ca
      link
      fedilink
      English
      arrow-up
      21
      ·
      14 days ago

      Because you are coming from the perspective of a reasonable person

      These people are billionaires who expect to get everything for free. Rules are for the plebs, just take it already

    • T156@lemmy.world
      link
      fedilink
      English
      arrow-up
      4
      ·
      edit-2
      14 days ago

      Because it takes work to obey the rules, and you get less data for it. The theoretical competitor could get more ignoring those and get some vague advantage for it.

      I’d not be surprised if the crawlers they used were bare-basic utilities set up to just grab everything without worrying about rules and the like.

  • AnthropomorphicCat@lemmy.world
    link
    fedilink
    English
    arrow-up
    32
    ·
    15 days ago

    So the world is now wasting energy and resources to generate AI content in order to combat AI crawlers, by making them waste more energy and resources. Great! 👍

    • brucethemoose@lemmy.world
      link
      fedilink
      English
      arrow-up
      11
      arrow-down
      4
      ·
      edit-2
      15 days ago

      The energy cost of inference is overstated. Small models, or “sparse” models like Deepseek are not expensive to run. Training is a one-time cost that still pales in comparison to, like, making aluminum.

      Doubly so once inference goes more on-device.

      Basically, only Altman and his tech bro acolytes want AI to be cost prohibitive so he can have a monopoly. Also, he’s full of shit, and everyone in the industry knows it.

      AI as it’s implemented has plenty of enshittification, but the energy cost is kinda a red herring.

  • TorJansen@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    27
    ·
    15 days ago

    And soon, the already AI-flooded net will be filled with so much nonsense that it becomes impossible for anyone to get some real work done. Sigh.

  • surph_ninja@lemmy.world
    link
    fedilink
    English
    arrow-up
    20
    ·
    14 days ago

    I’m imagining a sci-fi spin on this where AI generators are used to keep AI crawlers in a loop, and they accidentally end up creating some unique AI culture or relationship in the process.

  • Dr. Moose@lemmy.world
    link
    fedilink
    English
    arrow-up
    17
    arrow-down
    1
    ·
    edit-2
    15 days ago

    Considering how many false positives Cloudflare serves I see nothing but misery coming from this.

    • Xella@lemmy.world
      link
      fedilink
      English
      arrow-up
      3
      ·
      14 days ago

      Lol I work in healthcare and Cloudflare regularly blocks incoming electronic orders because the clinical notes “resemble” SQL injection. Nurses type all sorts of random stuff in their notes so there’s no managing that. Drives me insane!

  • XeroxCool@lemmy.world
    link
    fedilink
    English
    arrow-up
    13
    ·
    15 days ago

    Will this further fuck up the inaccurate nature of AI results? While I’m rooting against shitty AI usage, the general population is still trusting it and making results worse will, most likely, make people believe even more wrong stuff.

  • weremacaque@lemmy.world
    link
    fedilink
    English
    arrow-up
    12
    ·
    edit-2
    15 days ago

    You have Thirteen hours in which to solve the labyrinth before your baby AI becomes one of us, forever.