How to implement a custom sellable Items Asynchronous Strategy for Coveo Indexing

Sitecore Commerce: How to implement a custom sellable Items Asynchronous Strategy for Coveo Indexes

Yehia Ibrahim

I am a Senior production support developer at Velir. I Specialize in Sitecore technologies in the production support department.

Earlier this month, One of my clients reported an issue with products indexing. The issue was that when they create and publish new versions of the products , Sometimes the products appear in search results and sometimes they don’t. The other issue is that when a product appears in the search results, They were not able to add the product to the cart and they get 404 when they try to view the product detail page.

The products search results page uses Coveo and the product detail page and the ability to add the product to the cart are dependent on the Solr index.

A temporary fix for this issue was rebuilding both the Coveo and Solr Indexes which forces the products to be added to both indexes.

The root cause

The SellableItemsIntervalAsyncStrategyWeb strategy that is attached to the web index uses a list to retrieve sellable item changes that need to be updated once the interval (by default 10 minutes) is triggered. Once the product has been crawled, it will be removed from the list. The problem we were having is that we had the Coveo and Solr indexes use the same SellableItemsIntervalAsyncStrategyWeb Strategy. Both Sitecore_web_index and Coveo_web_index were using this strategy. This means that when the strategy runs for one of the indexes, it will crawl the product and update it on that index and remove it from the list, which means that it will not be available for the other index once it’s strategy triggers.

The resolution

To fix this issue, We needed to create our own custom Coveo strategies with new incremental index lists on the commerce engine for products to be added to incrementally.

Creating the Coveo Master and Web Index lists in the Commerce Engine

All indexing configuration is stored in the Plugin.CatalogIndexing.PolicySet-1.0.0.json file. You must add a new policy that instructs the Commerce Engine to create new lists for tracking incremental updates to your custom index.

To create new lists in Commerce Engine to track incremental changes:

  1. Navigate to the c:\inetpub\wwwroot\<environment>\wwwroot\data\Environments folder and open the Plugin.CatalogIndexing.PolicySet-1.0.0.json file.
  2. Under the "$values" list, add a policy block to create The Solr and Coveo managed lists for incremental updates to your custom index
  3. You will see that I created 2 more lists for Coveo in addition to the already created Sitecore index lists:
    1. SellableItemsCoveoIndexMaster: For coveo_master_index
    2. SellableItemsCoveoIndexWeb: For coveo_web_index
  4. Bootstrap the Commerce Engine to persist your changes.
  5. Perform an IIS reset in the IIS Manager.
"$type": "Sitecore.Commerce.Plugin.Catalog.SitecoreCatalogIndexingPolicy, Sitecore.Commerce.Plugin.Catalog",
"Name": "SellableItemsIndexMaster",
"IncrementalListName": "SellableItemsIncrementalIndexMaster",
"DeletedListName": "SellableItemsDeletedIndexMaster",
"EntityTypeNames": {
"$type": "System.Collections.Generic.List`1[[System.String, mscorlib]], mscorlib",
"$values": ["Sitecore.Commerce.Plugin.Catalog.SellableItem, Sitecore.Commerce.Plugin.Catalog"]},
"IsMasterIndex": true},

{"$type": "Sitecore.Commerce.Plugin.Catalog.SitecoreCatalogIndexingPolicy, Sitecore.Commerce.Plugin.Catalog",
"Name": "SellableItemsIndexWeb",
"IncrementalListName": "SellableItemsIncrementalIndexWeb",
"DeletedListName": "SellableItemsDeletedIndexWeb",
"EntityTypeNames": {
"$type": "System.Collections.Generic.List`1[[System.String, mscorlib]], mscorlib",
"$values": ["Sitecore.Commerce.Plugin.Catalog.SellableItem, Sitecore.Commerce.Plugin.Catalog"]},
"IsMasterIndex": false

"$type": "Sitecore.Commerce.Plugin.Catalog.SitecoreCatalogIndexingPolicy, Sitecore.Commerce.Plugin.Catalog",
"Name": "SellableItemsCoveoIndexMaster",
"IncrementalListName": "SellableItemsIncrementalCoveoIndexMaster",
"DeletedListName": "SellableItemsDeletedCoveoIndexMaster",
"EntityTypeNames": {
"$type": "System.Collections.Generic.List`1[[System.String, mscorlib]], mscorlib",
"$values": ["Sitecore.Commerce.Plugin.Catalog.SellableItem, Sitecore.Commerce.Plugin.Catalog"]},
"IsMasterIndex": true

"$type": "Sitecore.Commerce.Plugin.Catalog.SitecoreCatalogIndexingPolicy, Sitecore.Commerce.Plugin.Catalog",
"Name": "SellableItemsCoveoIndexWeb",
"IncrementalListName": "SellableItemsIncrementalCoveoIndexWeb",
"DeletedListName": "SellableItemsDeletedCoveoIndexWeb",
"EntityTypeNames": {
"$type": "System.Collections.Generic.List`1[[System.String, mscorlib]], mscorlib",
"$values": ["Sitecore.Commerce.Plugin.Catalog.SellableItem, Sitecore.Commerce.Plugin.Catalog"]},
"IsMasterIndex": false

Adding the new Coveo Sellable Items Index Update Strategies

The next step is to create Coveo Master and Web Index Update Strategies that will use the 2 new lists we created in the commerce engine.

Under the IndexUpdateStrategies node in the configuration, Add the following Block.

As you can see in the code block below, The sellableItemsIntervalAsynchronousStrategyCoveoMaster uses the SellableItemsIncrementalCoveoIndexMaster and SellableItemsDeletedCoveoIndexMaster that we created in the previous step. And the sellableItemsIntervalAsynchronousStrategyCoveoWeb uses SellableItemsIncrementalCoveoIndexWeb and SellableItemsDeletedCoveoIndexWeb

          <sellableItemsIntervalAsynchronousStrategyCoveoMaster role:require="Standalone or ContentManagement" type="Sitecore.Commerce.Engine.Connect.Search.Strategies.SellableItemsIntervalAsynchronousStrategy, Sitecore.Commerce.Engine.Connect">
            <Environments hint="list">
            <param desc="interval">00:10:00</param>
            <param desc="database">master</param>
          <sellableItemsIntervalAsynchronousStrategyCoveoWeb type="Sitecore.Commerce.Engine.Connect.Search.Strategies.SellableItemsIntervalAsynchronousStrategy, Sitecore.Commerce.Engine.Connect">
            <Environments hint="list">
            <param desc="interval">00:10:00</param>
            <param desc="database">web</param>

Update the Coveo Search provider config to point to the new Strategies

In the Coveo Search provider Configurations, Make sure you update the Strategies references to point to the newly created strategies instead of the default Sitecore Sellable Items Strategies.

<strategies role:require="!ContentDelivery" hint="list:AddStrategy">
    <strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/sellableItemsIntervalAsynchronousStrategyWeb" />
    <strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/sellableItemsIntervalAsynchronousStrategyCoveoWeb" 
<strategies hint="list:AddStrategy">
    <strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/intervalAsyncMaster"/>
    <strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/sellableItemsIntervalAsynchronousStrategyCoveoMaster"/>

After Completing the Steps Above, You should now be able to see the Sellable Items being indexed in both Sitecore and Coveo Indexes.

One issue encountered

After deploying this fix to the UAT environment, I created and published a new product from the Sitecore Business tools and I verified that the product exists in the Coveo and Sitecore Indexes.

However, When I checked the website. The product showed up in the Coveo search result page but I was still getting the 404 when I view the product detail page and also I was still not able to add the product to the cart. So it looked like the caches are not properly being cleared for the web DB but they are for master DB.

After investigating, I found that the issue is that the event queue provider for the CM and CD are different. On CM the core DB is used while on CD the web DB is used.
This means that CD will not receive any of the remote events that have been triggered on CM, this includes the indexing:end:remote event which is needed for web caches to be cleared.
So I added the ContentManagement role to your CM and CD instances under the following node and that resolved the caching issue.

<eventQueueProvider role:require="ContentManagement or ContentDelivery">
<patch:attribute name="defaultEventQueue">web</patch:attribute>

Thank you for reading!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: