I built NextOnMenu, a little early-signal model for which food ingredient might pop off next (matcha, tahini, yuzu…). The idea: early on a trend's search interest is noisy and random, so high entropy. Right before it breaks out, the signal organizes into a regular band and entropy drops. Then the mainstream spike comes.

    Top: search interest. Bottom: rolling Shannon entropy. The dashed line is the mainstream peak; the entropy bottoms out weeks before it.

    Made with a little entropy library I wrote, entroscope. The whole chart is

    shannon.rolling(s, window=10)

    (Synthetic signal illustrating the pattern. Tools: Python, entroscope, matplotlib.)

    entroscope: https://github.com/Par-python/entroscope

    Posted by jRetro3

    Share.

    6 Comments

    1. Why is there a month long plateau in your synthetic signal at 2024-03? A plateau will manifest itself as a valley in the entropy.

      Plug in real data and you’ll likely get noise.

    2. revolutionofthemind on

      Do you know how the search interest data is collected? Have you found any search terms that have stabilized without picking up in interest? I’m wondering if this is an artifact of data collection and presentation (e.g. more data points averaged in high interest periods) vs something detectable in advance

    3. is this just because of the stable plateau before? Variance goes down, therefore entropy goes down too.

      Maybe this is the usual case and able to predict what might become mainstream, but with only one ingredient, this is not good data. You need to show the time course of more ingredients to really make this believable.

    4. >Right before it breaks out, the signal organizes into a regular band and entropy drops

      But why does it do that?