• [object Object]@lemmy.ca
    link
    fedilink
    English
    arrow-up
    13
    ·
    edit-2
    5 hours ago

    I feel vindicated on the number of times I’ve had to tell managers you don’t need optimizely, you can run a t-test super simply, and you can manage your type I error via simulations also easily.

    For sequential testing like these online systems do there are a lot of options, but I usually direct them to Bayesian methods as they can show the uncertainty in a clearer way.

    But basically, bootstrapping plus Bayes equals pretty good interpretable metrics.

    Like, before LLMs I could spin up a tool for this in a day or two.

    See also https://www.evanmiller.org/index.html for some good content on this, so you can convince your companies to not use optimizely.

    If you want to learn more read Computer Age Statustical Inference by Effron and Haste. You won’t get anything better than Effron’s writing on the topic.

    ETA: it was like many many years before Optimizely actually hired statisticians to fix their awful statistical methods, and even then, they published some papers on stopping time that I think were mostly overcomplicated and for marketing reasons.

    The culture to me from the outside seems like it’s long been about metrics to make product managers fork over cash instead of the making optimal decisions.