Sitecore 8 Upgrade: Lessons Learned

In my first Sitecore 8 Upgrade use case, described in my Upgrade from Sitecore 6.6 to Sitecore 8 series, I learnt several valuable lessons along the way. I will not be able to spell all those out for you. Therefore I decided to share a summary of those high level points with you. Hope you will be kind enough as well to add to this list. After all as a Sitecorian, sharing is caring.

Lesson 1: Addressing your Customer Requirements well.

Having well addressed all key success criterias related for my Customer, I feel that sometimes we may not address the requirements well if we do not understand what is needed to resolve the problem.

The key thing is to understand your Customer’s environmental variables. Next, is to ensure that you are confident with your proposed solution. This comes with much planning and analysis on the system’s problematic points and how the upgrade can address them.

If you are new to Sitecore, I will strongly recommend reading up Sitecore MVP blogs and visit Sitecore Community forums to speed up your learning. Having said that, nothing beats going through the original Sitecore Documentation in SDN and Docs Sitecore to understand basic and fundamentals.

Communicate with your client often to gather information during the analysis phase. In the first series, I explained about the must ask questions to your Customer when performing an upgrade. Use this as a guideline to help you make good decisions on your proposed solution.

Build a presentation deck to describe the scope of your work and how they will address your customer’s requirements, whether it will be performance issues, new features etc. This can better align yours and your customer’s understanding on the subject matter when the you are presenting your deck before the customer. This promotes a Q&A session after the presentation to better clarify requirements.

Lesson 2: Never gold plate your Customer Requirements.

Keep things simple. Ensure requirements are met to baseline standards. This can even save you from spending long hours of development whilst minimising the risk of your inability to deliver to your customer on time and on budget.

Putting your Customer’s main needs was the driving factor for the project success!

Lesson 3: If your customers are not familiar with Sitecore’s new CMS UI Search, show it to them earlier before the project kick off.

Sitecore 6.x does not feature any ability to search content or media using keywords, tags or facets.

Therefore when required, demo the use of Sitecore 8 much early in the phase to demonstrate how their users will be comfortable to use new search features. This will be their daily bread and butter and of course, and they must like the system and be comfortable to use it. This is especially important to prevent later surprises during User Acceptance Testing or Training sessions, whereby the user may not be happy with certain features. Get an agreement much earlier on to negotiate with basic features on the product and put this down into the requirements.

We realised we managed obtain just two Change Requirements pertaining to User Experience just after the User Acceptance stage. The above saved my team a lot of effort.

Lesson 3: My index rebuild is timing out and taking an extremely long time. Thank you dear Sitecore Logs. 

Turn off Analytics if you do not need it.

Thanks to our best friend, Sitecore Logs, I have managed to pinpoint the root cause of the issue.

If you are not working with Analytics, turn it off. Yes, it is simple to say that. However, it is not straightforward in Sitecore 8.0 with a flip of a switch (i.e it is not just only disabling a configuration setting in your .config). More info will be posted in my article about this.

Turning off Analytics reduced index rebuild time by two hours.

The reason why we needed to turn it off was due to the fact that a full index rebuild of the Master database took long hours to rebuild and timed out at the final step of index optimisation. This optimisation operation is performed by Solr prior to finalising and preparing its index for query readiness.

Turn off Item Cloning

This can be done by setting the ItemCloning.Enabled config property to false. This is  a gold configuration that not many people know. But before you turn it off, be sure to know what the configuration does. Disabling this property can can reduce memory consumption substantially and the reduce number of SQL calls to the Master database. We realised after this was disabled, turning off this setting reduced index rebuild time by a further four hours.

Increase your DI Container Timeout

We used Solr Cloud as our Search provider with Sitecore. By using Solr, we needed to choose a DI Container. By increasing the timeout on our DI container, this helped us to extend the time taken to rebuild and optimise the full Master index. By default, Sitecore sets this to a 30 minute timeout. This timeout value can be set by overridding the Application_Start method in the Global.asax.

Lesson 4: Use console applications to run long running processes.

This is especially true for the Data Migration implementation. We chose to use Console Applications to develop our Migrator Agents to migrate thousands of news, articles and media definition items to designated Sitecore Buckets.

Although Sitecore does feature Background Jobs,  however in our case, we were primarily concerned with web timeout operations. Using a console app separates all our long running activities from the web context and provided us assurance of non time out issues. This is especially true for large batches of data to migrate, especially with media definition items that carried over large binary data. Moreover, building with a Console Application gave us better control of code customisation and ensure a smooth and reliable migration process.

The above approach helped us to complete all data migration activities before the specified deadline.

Lesson 5: Make a back up of your data just before you execute your migration process.

Can’t emphasise this more. This is especially true if you mess up with data. We had an issue whereby we did not have a point in time backup to restore to after our Media Migrator Agent accidentally migrated Sitecore’s default system images to our designated custom Media Folder Bucket. We needed to then restore to current morning’s data and repeat a few complex activities prior to being able to run the agent again.

There are even possible occasions that you will need to repeat the migration process several times to test different scenarios. Backup your data in case!

Lesson 6: Data migration may slow down tremendously after a few hours.

This was due to the fact that Sitecore creates an event record on each content or definition item that is moved during migration. In turn, this creates heavy write operations to the Master database, that in turns causes the migration process to progressively decrease resulting in poor slow down.

Execute period cleaning of your EventQueue and PublishQueue tables to prevent Publishing Queue. Perform IIS restart to ensure that all processes and blocking threads are flushed prior to resuming the migration agent.

Lesson 7: Sitecore’s Content Management server eats memory like a beast

As we were performing a full index rebuild, we realised that IIS instance memory for our Sitecore solution started to grow as the index rebuild progressed along. However, with some of the extra activities turned off in our Sitecore configuration, this reduced slight memory consumption on the server. Having said that, Sitecore still does consume memory when it attempts to cache the items that are indexed into memory.

The solution was to increase memory capacity on the CM server to accomodate the index rebuild.

Lesson 8: Place higher CPU for your Solr Servers if using Solr Cloud

We implemented Solr Cloud for our customer whereby each Solr server will have an exact copy of our out of the box indexes and our custom indexes. Three Solr Servers were setup in a minimal quorum for leader election among the participating nodes (also known as followers) should one of the servers go down during an unexpected outage or if a service interruption occurred on one of the nodes. This setup is mandatory for minimal high data availability.

As we were monitoring the Solr Servers for the first week of the Post Go-Live period, we realised that CPU levels were peaking intermittently due to periods where large traffic spikes occurred and also during intervals where large batches of articles were created during a specified time of the day. This caused poor page loading of the home landing page at intermittent periods.

We realised that Solr services on the three servers consumed high CPU resources to replicate its indexes across its followers. Solr Replication is deemed an expensive operation. The resolution was to increase CPU capacity to accommodate the replication work load.

Hope that this article does help you to achieve your upgrade. Do note that every upgrade will have varying requirements and traps. The explanation above is solely a guide and what I can share with you so far with my experience.

If there is anything you will like to add or remark, please feel free to comment and share on my blog. It will be as equally exciting to hear what others have learnt in your Sitecore upgrade projects.

 

2 thoughts on “Sitecore 8 Upgrade: Lessons Learned

  1. Adrian,
    I see you have a great amount of experience with Sitecore, and I also see you have knowledge of SOLR.
    We are currently having a tone of issues with our SOLR implementation and Sitecore. We do not use SOLR for site search, for that we use Coveo. However, our analytics index (In SOLR)is constantly failing. We have the famous “exceeded limit of maxWarmingSearchers=2” errors.

    Our index is roughly 11.5 Gbs, we have 3 SOLR server configuration with Zookeeper. Each box has 4 cores with 16 G of RAM with 8 allocated for the JVM. (I thought that will be enough)

    Have you ever come across anything similar? And if so, how did you address it?

    1. Hi Camilo,
      Very sorry for the 4 year late reply! I have seen something similar appearing in our Solr logs at that point of time during my Sitecore implementation. The solution was to tune your working memory set in Solr and ensure they are performing to the size of your index and to additionally tweak the TimeIntervalCommitPolicy in your Solr configuration. Please also make sure your Solr instance is running on 64 bit environment. Additionally, a good spec-ed Solr machine / app service with a good RAM juice at least 8GB RAM is recommended for a production setup. It may be abit too late to address it now as maybe you have performed an upgrade already to Sitecore 9? If you already do, I will recommend offloading Solr to an external provider such as SearchStax. As SearchStax provides vanilla configuration that are preset with the optimal configurations for Sitecore 9.x.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.