Insights

Top selling items on Amazon included tech till early April and toiletries later on. In this abrupt U-turn of customer preferences, the recommender systems went berserk – the popularity of e-commerce broke all records, achieving 10 years’ of growth in just eight weeks.

A disgruntled mask broker called the market a “madhouse”. Others opined it was “the craziest market they’d ever seen”.

New item stars made their debut almost every hour. Bulk toilet paper jumped from rank 2,657 to rank 4. One item – “a pack of 100 masks” – lunged 37,181 ranks to make it into the top 10.

While this may appear catastrophic to some, the foresighted ones swallowed it up as the red pill.

COVID shook up systems. It jolted our recommender engines in a way never seen before. The stockpiling confidence of a decade of successes came crashing down like a house of cards.

Many machine learning models trained to address normal human behaviors no longer worked as they should. Others realigned their definition of normal.

The pandemic served as an eye-opener for the engineers who boasted about the singularity of algorithms and consigned data to the back-burner. In the aftermath of the chaos, a collective consensus grew – ML models perform badly when fed with data that runs askew from their training data.

There was a silver line to this dark cloud – data science professionals acquired the recognition they deserved. In a recent interview, Rael Cline, the CEO of Nozzle, said:

“You need a data science team who can connect what’s going on in the world to what’s going on the algorithms. An algorithm would never pick some of this stuff up.”

Nozzle specializes in algorithmic advertising for amazon sellers.

Now that Data Science professionals have entrenched their mark, how do they navigate this new normal?

“The world has changed, and the data has changed.” – David Excell, founder of Featurespace, a behavior analytics company

In this altered reality, recommender systems are expected to be exceedingly data- or knowledge-driven and will be rooted in the customer’s cognitive aspects like behavior, personality, and attitude.

The future of recommendation systems will be driven by these five trends:

Personal preferences will take the spotlight in the coming days. Netflix is already incorporating the same in myriad ways.

“We have discovered through the years that there is tremendous value to our subscribers in incorporating recommendations to personalize as much of Netflix as possible.” –
Xavier Amatriain and Justin Basilico

SAP’s new Recommendation Service API draws “insights on user patterns from historical user behavior (e.g. clickstream) as well as available metadata (e.g. item catalog, user profile)”, delivering highly personalized recommendations. This means multiple users checking the same item will get different results.

Customer confidence has dipped: they are postponing big decisions like where to spend holidays and live. And health, meaningfulness, and comfort have become a priority. They are reluctant to spend in new places.

Building data for long-term and costly purchases will remain a challenge.

Before COVID, “viral” was a trending tag. Not anymore. Any word that indicates a link to COVID can be a big turn-off.

London-based Phrasee, a copywriting tech firm serving big brands like eBay, Groupon, and Sephora, had to filter out the language in its algorithm and remove phrases like “going viral”, “party wear”, “OMG”, “stock up”, “be prepared”, and “brace yourself”.

The market is volatile. One week you are selling toilet papers and the next week you are channeling gym items. Rajeev Sharma, global vice president at Pactera Edge, a data, intelligence and experience provider, says - ML models, in addition to regular ups and downs, must be trained on freak events like the Great Depression of the 1930s, the Great Recession of 2008, and the Black Monday stock market crash of 1987.

Delivery time has taken precedence, and if you aren’t sure that you’ll be able to deliver faster than your competitor, then you may as well not try. New factors are impacting purchase decisions: multiple pages and modules, multiple sessions, and multiple devices.

The number of passive and spontaneous customers is steadily growing. Impulse purchases are surging, and an increasing number of customers are displaying non-repetitive behaviors.

Data Science professionals also need to see how they cut the growing noise around data.

Recommendation systems are an important cog in the ROI wheel. They:

• Generate 35% of Amazon’s revenue (McKinsey & Company)
• Keep 75% of user traffic on Netflix (McKinsey & Company)
• Help Spotify keep its 100 million user base (Spotify)
• Account for 60% of video clicks on YouTube’s Homepage (YouTube)

COVID has altered customer behavior. The way they interact online has changed, and recommender systems must upgrade accordingly.

To get the best results, Data Science professionals must understand the entire ecosystem, including the interplay between the algorithms and search modules. Unless we optimize user engagement and create a coherent user experience, we won’t be able to achieve much from them.

We part for now. Don’t forget to check out the basics of the recommender system we have covered here.

Recommender systems have been expanding. Take a look at this sprawling umbrella.