Featured

How to implement a custom sellable Items Asynchronous Strategy for Coveo Indexing

Sitecore Commerce: How to implement a custom sellable Items Asynchronous Strategy for Coveo Indexes

Yehia Ibrahim

I am a Senior production support developer at Velir. I Specialize in Sitecore technologies in the production support department.

yehiasaad88@gmail.com

Earlier this month, One of my clients reported an issue with products indexing. The issue was that when they create and publish new versions of the products , Sometimes the products appear in search results and sometimes they don’t. The other issue is that when a product appears in the search results, They were not able to add the product to the cart and they get 404 when they try to view the product detail page.

The products search results page uses Coveo and the product detail page and the ability to add the product to the cart are dependent on the Solr index.

A temporary fix for this issue was rebuilding both the Coveo and Solr Indexes which forces the products to be added to both indexes.

The root cause

The SellableItemsIntervalAsyncStrategyWeb strategy that is attached to the web index uses a list to retrieve sellable item changes that need to be updated once the interval (by default 10 minutes) is triggered. Once the product has been crawled, it will be removed from the list. The problem we were having is that we had the Coveo and Solr indexes use the same SellableItemsIntervalAsyncStrategyWeb Strategy. Both Sitecore_web_index and Coveo_web_index were using this strategy. This means that when the strategy runs for one of the indexes, it will crawl the product and update it on that index and remove it from the list, which means that it will not be available for the other index once it’s strategy triggers.

The resolution

To fix this issue, We needed to create our own custom Coveo strategies with new incremental index lists on the commerce engine for products to be added to incrementally.

Creating the Coveo Master and Web Index lists in the Commerce Engine

All indexing configuration is stored in the Plugin.CatalogIndexing.PolicySet-1.0.0.json file. You must add a new policy that instructs the Commerce Engine to create new lists for tracking incremental updates to your custom index.

To create new lists in Commerce Engine to track incremental changes:

  1. Navigate to the c:\inetpub\wwwroot\<environment>\wwwroot\data\Environments folder and open the Plugin.CatalogIndexing.PolicySet-1.0.0.json file.
  2. Under the "$values" list, add a policy block to create The Solr and Coveo managed lists for incremental updates to your custom index
  3. You will see that I created 2 more lists for Coveo in addition to the already created Sitecore index lists:
    1. SellableItemsCoveoIndexMaster: For coveo_master_index
    2. SellableItemsCoveoIndexWeb: For coveo_web_index
  4. Bootstrap the Commerce Engine to persist your changes.
  5. Perform an IIS reset in the IIS Manager.
{
"$type": "Sitecore.Commerce.Plugin.Catalog.SitecoreCatalogIndexingPolicy, Sitecore.Commerce.Plugin.Catalog",
"Name": "SellableItemsIndexMaster",
"IncrementalListName": "SellableItemsIncrementalIndexMaster",
"DeletedListName": "SellableItemsDeletedIndexMaster",
"EntityTypeNames": {
"$type": "System.Collections.Generic.List`1[[System.String, mscorlib]], mscorlib",
"$values": ["Sitecore.Commerce.Plugin.Catalog.SellableItem, Sitecore.Commerce.Plugin.Catalog"]},
"IsMasterIndex": true},

{"$type": "Sitecore.Commerce.Plugin.Catalog.SitecoreCatalogIndexingPolicy, Sitecore.Commerce.Plugin.Catalog",
"Name": "SellableItemsIndexWeb",
"IncrementalListName": "SellableItemsIncrementalIndexWeb",
"DeletedListName": "SellableItemsDeletedIndexWeb",
"EntityTypeNames": {
"$type": "System.Collections.Generic.List`1[[System.String, mscorlib]], mscorlib",
"$values": ["Sitecore.Commerce.Plugin.Catalog.SellableItem, Sitecore.Commerce.Plugin.Catalog"]},
"IsMasterIndex": false
},

{
"$type": "Sitecore.Commerce.Plugin.Catalog.SitecoreCatalogIndexingPolicy, Sitecore.Commerce.Plugin.Catalog",
"Name": "SellableItemsCoveoIndexMaster",
"IncrementalListName": "SellableItemsIncrementalCoveoIndexMaster",
"DeletedListName": "SellableItemsDeletedCoveoIndexMaster",
"EntityTypeNames": {
"$type": "System.Collections.Generic.List`1[[System.String, mscorlib]], mscorlib",
"$values": ["Sitecore.Commerce.Plugin.Catalog.SellableItem, Sitecore.Commerce.Plugin.Catalog"]},
"IsMasterIndex": true
},

{
"$type": "Sitecore.Commerce.Plugin.Catalog.SitecoreCatalogIndexingPolicy, Sitecore.Commerce.Plugin.Catalog",
"Name": "SellableItemsCoveoIndexWeb",
"IncrementalListName": "SellableItemsIncrementalCoveoIndexWeb",
"DeletedListName": "SellableItemsDeletedCoveoIndexWeb",
"EntityTypeNames": {
"$type": "System.Collections.Generic.List`1[[System.String, mscorlib]], mscorlib",
"$values": ["Sitecore.Commerce.Plugin.Catalog.SellableItem, Sitecore.Commerce.Plugin.Catalog"]},
"IsMasterIndex": false
},

Adding the new Coveo Sellable Items Index Update Strategies

The next step is to create Coveo Master and Web Index Update Strategies that will use the 2 new lists we created in the commerce engine.

Under the IndexUpdateStrategies node in the configuration, Add the following Block.

As you can see in the code block below, The sellableItemsIntervalAsynchronousStrategyCoveoMaster uses the SellableItemsIncrementalCoveoIndexMaster and SellableItemsDeletedCoveoIndexMaster that we created in the previous step. And the sellableItemsIntervalAsynchronousStrategyCoveoWeb uses SellableItemsIncrementalCoveoIndexWeb and SellableItemsDeletedCoveoIndexWeb

          <sellableItemsIntervalAsynchronousStrategyCoveoMaster role:require="Standalone or ContentManagement" type="Sitecore.Commerce.Engine.Connect.Search.Strategies.SellableItemsIntervalAsynchronousStrategy, Sitecore.Commerce.Engine.Connect">
            <IncrementalIndexListName>SellableItemsIncrementalCoveoIndexMaster</IncrementalIndexListName>
            <DeletedIndexListName>SellableItemsDeletedCoveoIndexMaster</DeletedIndexListName>
            <ItemsToTake>100</ItemsToTake>
            <Environments hint="list">
              <environment>Authoring</environment>
            </Environments>
            <param desc="interval">00:10:00</param>
            <param desc="database">master</param>
          </sellableItemsIntervalAsynchronousStrategyCoveoMaster>
          <sellableItemsIntervalAsynchronousStrategyCoveoWeb type="Sitecore.Commerce.Engine.Connect.Search.Strategies.SellableItemsIntervalAsynchronousStrategy, Sitecore.Commerce.Engine.Connect">
            <IncrementalIndexListName>SellableItemsIncrementalCoveoIndexWeb</IncrementalIndexListName>
            <DeletedIndexListName>SellableItemsDeletedCoveoIndexWeb</DeletedIndexListName>
            <ItemsToTake>100</ItemsToTake>
            <Environments hint="list">
              <environment>Shops</environment>
            </Environments>
            <param desc="interval">00:10:00</param>
            <param desc="database">web</param>
          </sellableItemsIntervalAsynchronousStrategyCoveoWeb>

Update the Coveo Search provider config to point to the new Strategies

In the Coveo Search provider Configurations, Make sure you update the Strategies references to point to the newly created strategies instead of the default Sitecore Sellable Items Strategies.

<strategies role:require="!ContentDelivery" hint="list:AddStrategy">
    <strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/sellableItemsIntervalAsynchronousStrategyWeb" />
    <strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/sellableItemsIntervalAsynchronousStrategyCoveoWeb" 
</strategies>
<strategies hint="list:AddStrategy">
    <strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/intervalAsyncMaster"/>
    <strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/sellableItemsIntervalAsynchronousStrategyCoveoMaster"/>
</strategies>

After Completing the Steps Above, You should now be able to see the Sellable Items being indexed in both Sitecore and Coveo Indexes.

One issue encountered

After deploying this fix to the UAT environment, I created and published a new product from the Sitecore Business tools and I verified that the product exists in the Coveo and Sitecore Indexes.

However, When I checked the website. The product showed up in the Coveo search result page but I was still getting the 404 when I view the product detail page and also I was still not able to add the product to the cart. So it looked like the caches are not properly being cleared for the web DB but they are for master DB.

After investigating, I found that the issue is that the event queue provider for the CM and CD are different. On CM the core DB is used while on CD the web DB is used.
This means that CD will not receive any of the remote events that have been triggered on CM, this includes the indexing:end:remote event which is needed for web caches to be cleared.
So I added the ContentManagement role to your CM and CD instances under the following node and that resolved the caching issue.

<eventQueueProvider role:require="ContentManagement or ContentDelivery">
<patch:attribute name="defaultEventQueue">web</patch:attribute>
</eventQueueProvider>

Thank you for reading!

A PowerShell script to fix broken renderings in Content Pages

Sitecore PowerShell: A PowerShell script to fix broken renderings in Content Pages

Yehia Ibrahim

I am a Senior production support developer at Velir. I Specialize in Sitecore technologies in the production support department.

yehiasaad88@gmail.com

One of my clients reported an issue that when they try to edit and then save a page in the Sitecore content tree, they get an error message saying that the page contains broken links in the final renderings field.

Some of the renderings in the branch template were pointing to the wrong data source. So with every new page they create they get this error message upon saving the page. I fixed the datasource in the branch template but the problem is that they had already created a lot of pages before fixing the branch template.

Fixing the existing pages

To fix the existing pages, I created a Sitecore PowerShell script to iterate over the pages and check for broken/Invalid links in the data source field of every rendering on the page.

The script:

  1. We had thousands of pages in the content with broken final renderings so I used the sitecore master index in the query instead of running a regular sitecore query (line 1)
  2. I applied 2 filters. one for the template and the other one is the start location of the search. You can tweak these based on what content is affected (line 5 and 10).
  3. Iterate over all the renderings and check if the datasource is valid. if getting the dataource item returned null then I replace the dataource field in the rendering with an empty/null value (line25).
  4. $count variable is just to display how many renderings have been fixed after completing running the script.
$results = Find-Item -Index sitecore_master_index -Criteria @(
@{
	Filter = "Equals";
	Field = "_templatename";
	Value = "Detail Page"    #Specify the template name here
}
@{
    Filter = "StartsWith";
    Field = "_fullpath";
    Value = "/sitecore/content"}  #specify the start location here
) | Initialize-Item

$count =0;

$results | ForEach-Object {
    
    $renderingInstance = Get-Rendering -Item $_ -FinalLayout -Rendering $rendering
    
            for($i=0; $i -lt $renderingInstance.Length; $i++){
            $currentID = $renderingInstance[$i].datasource
                    if($currentID){
                        $currentItem = Get-Item -Path master: -ID $currentID
                         if(!$currentItem){
                            $count++;
                            $renderingInstance[$i].datasource = $null
                            Set-Rendering -Item $_ -Instance $renderingInstance[$i] -FinalLayout
                          }
                    }
            
            }
    
    
}
Write-Host $count

Thank you for reading!