Archive for the ‘SharePoint Las Vegas 2012’ Category

Shredded Storage: A new feature in SharePoint 2013

December 7th, 2012 Comments off

Shredded Storage is a new feature introduced in SharePoint 2013. It deals with the way that SharePoint stores documents/files in Binary Large Objects (BLOBs) in SQL Server to improve both storage utilization and I/O performance.

Although ‘shredded’ sounds a bit ominous, it means that SharePoint now turns each file into a number of shreds and writes these shreds to SQL as separate BLOBs. It includes an index that keeps track how the various shreds fit together to make sure that a file can be reassembled when it is requested.

Now this in itself will obviously not improve storage or I/O. However, let’s go in a bit more depth by looking at a document library in SharePoint which has versioning enabled.

In SharePoint 2010, updating a file in a version enabled library created a new version record for that file including a new BLOB which held the entire updated file. Updating meta-data information such as title or contact or any other field was also considered to result in a new version.  SharePoint would then again create another BLOB for that file even though the file itself was not changed.

So if you had a 1MB file in a versioned library for which the meta-data was changed 9 times, 10 identical BLOBs with a total of 10MB would be stored in SQL, one for the latest version and 9 for the previous versions.

The way shredded storage improves on this process when updating files is by working only with those shreds that actually changed. In case of a versioned library, a new version record is still created with the latest meta-data however the only BLOBs that will be added are the ones that correspond to those shreds that actually changed. The shred-index for this version record which is needed to recreate the full file will be a combination of entries that point to the unchanged shreds of the previous version(s) and the entries that point to the newly added changed shreds. In case of an update that only includes meta-data changes, the number of changed shreds is basically zero which means that no new BLOBs are written to SQL and the shred-index for the new version is the same as the previous one.

As you can imagine this could dramatically reduce storage for versioned document libraries and also improve the I/O performance as only the changed shreds have to be written to SQL. Because of the way Microsoft Office documents are now structured (XML) and the use of a technology called Cobalt (introduced in SharePoint 2010 to make sure that only changes are sent when editing documents on SharePoint directly in Microsoft Office), Microsoft Office documents will benefit even more from the new shredded storage technology.

Another thing you should know about shredded storage is that it can be enabled (default) or disabled on a web application level. The desired setting can be chosen by manipulating the FileOperationSettings property of the WebService member of the web application (

This property has 3 possible settings:

  • UseWebSetting (=0)
  • AlwaysDirectToShredded (=1)
  • NeverDirectToShredded (=2)

The FileWriteChunkSize property can subsequently be used to change the maximum size that will be used for each shred (default=64kB). (

As the first setting for FileOperationSettings suggests, it should also be possible to control the setting on a per sub site basis. This would be done by manipulating the EnableDirectToShreddedStorage property of the SPWeb class but unfortunately this is currently not (yet) documented at

Something else that is worth mentioning is that shredded storage is a per document feature. So if two exactly the same documents are stored in two different libraries, these two documents will still have their own set of shreds which will take up twice the space of each individual document.
The only solution that would actually save on storage for these kinds of situations is to enable Remote BLOB Storage (RBS) using an implementation that supports de-duplication. This way BLOBs are stored outside of SQL on a file system and the RBS layer will make sure that identical BLOBs only take up space once. If shredded storage is enabled in combination with RBS, each shred is stored on the file system as a separate BLOB. If the chosen RBS solution supports de-duplication, identical shreds will only take up space once, even if they belong to different files in SharePoint.

Although shredded storage is enabled by default, it only applies to documents from the first time they are uploaded or changed. For a new site this would mean that all new documents will be stored as shreds. However when you upgrade a site, the already existing files will not be stored as shreds during the upgrade. Only when the existing files are changed, they will be ‘shredded’.

Based on the information discussed so far, the conclusion would be that shredded storage potentially lowers the amount of storage required for storing files that are available in SharePoint. On the subject of I/O performance reduction, Cobalt (introduced in SharePoint 2010) takes care of reducing the amount of data that is sent when users update Microsoft office documents in Office 2010. Shredded storage adds to the I/O reduction by reducing the amount of data that is sent to the SQL server when users update documents in SharePoint.

But what about read I/O performance?

Because shredded storage breaks files in multiple chunks when they are added to SharePoint, reading such a file would be more complex as SharePoint needs to get the correct set of shreds to be able to assemble the original file which should take longer than reading the entire file at once as was the custom in SharePoint 2010.

During the SharePoint Conference 2012 in Las Vegas, Dan Holme (Intelliem) and Jeremy Thake (AvePoint) presented some test results with regards to the impact of shredded storage on read I/O (and data storage). For the test a SharePoint site was used that contained 24GB of files. Tests were done with and without the use of RBS. For the base test, both shredded storage and RBS were disabled. In this case the database size was around 24GB and the RBS storage size was 0GB. Read time for this base test was measured and amounted up to 1477ms. After RBS was turned on, the database size shrunk to a couple of MBs and the RBS storage size was about 23GB. Read times with the added RBS layer increased with approx. 25% to 1882ms.
The next test was done with shredded storage enabled (default chunk size of 64KB) and RBS disabled again. In this case the database shrunk with approx. 75% to 6GB. Read times increased to 2471ms which is an increase of approx. 67% compared to the base situation where shredded storage and RBS were disabled.

After RBS was enabled for this situation, database size shrunk to a some MBs again (although a bit more than with the first test) and RBS storage size was about 6GB. In this case read times again increased to 3502ms which is approx. 86% longer than the test without shredded storage and even 137% longer than the base test (no shreds & no RBS).
As a last step, the same situations were tested with larger chunk sizes (1MB and 1GB). This obviously reduces the number of shreds that need to be gathered for each document which resulted in better read times but they never came down to the 1477ms read time that was the result of the base test.

What the above described tests confirm is that shredded storage can definitely impact the amount of required storage in a positive way. Based on the rest of the story, it should also impact write I/O in a positive way. However with regards to getting data out of SharePoint, there’s an increase in the response times.

So all in all shredded storage is a new feature that could definitely be beneficial to SharePoint but not necessarily at all times so, as with all things, it depends on what you are looking for. For instance a site where most or all of the files are read-only might not be a good candidate for shredded storage. Also a site with lots of small files might not benefit that much. Whatever situation seems best for your particular situation, always make sure that you test it to verify whether you get the expected results.

The blessing of the app-model in SharePoint 2013

November 23rd, 2012 Comments off

Last week, we as Informed Consulting, attended the SharePoint Conference 2012 in Las Vegas and we all came back to our cold little country with lots of information and lots to think about.

But first, what a week! For me, this was my first big conference and the first time in Las Vegas, and it was a combination I will never forget. Absolute brilliant organisation by Microsoft, providing over 10.000 people, hungry for food and information, with their tasty meals, ready in no more than 10 minutes and so many interesting sessions to choose from that you are obligated to watch the sessions online that you could not attend at the conference itself!

When Microsoft launched the beta of SharePoint 2013, the biggest new feature for me was the whole app-model that was introduced.
But unfortunately, I could not get it working on my development machine, mainly because I did not have enough internal memory available to smoothly run the SharePoint 2013 VM.
Also, I did not get the whole app idea. Okay, it is cool to have a store where everybody could sell their apps, but how many corporate companies would allow users to install 3rd party apps on their corporate intranet? None I think, so why should we as developers stop creating WebParts and start building apps?

With that question in the back of my mind, I scheduled as many interesting sessions about apps in SharePoint 2013 as I could, hoping to get back with a well-founded answer.

And YES, Microsoft has made a huge step forward with the introduction of the SharePoint 2013 app model!
Apps in SharePoint 2013 are not about selling and distributing them in the appstore, it is all about creating functionalities on top of SharePoint that work just as good on premise as in Office 365 in the cloud. And also important: do not break SharePoint.

The current version of Office 365 is based on SharePoint 2010, and allows developers to create only SandBoxed WebParts. The big limitation of SandBoxed WebParts is not having the ability to run code with elevated privileges. This is something we do quite often when we build WebParts for corporate clients that have their SharePoint environment hosted on premise.
Because of this limitation, we have recommended a lot of our current clients not to move to the cloud. Losing all the custom built WebParts and functionalities that use elevated privileges, does not outweigh the advantages of hosting your SharePoint environment in the Cloud. But all of this is about to change with the new app model of SharePoint 2013!

To avoid getting into too much depth about the technical side of the app model, I will try to explain it at a high level.

The code that you would develop for the previous versions of SharePoint all run on the SharePoint server itself. You always had to make sure that your code was written properly and it did not have for example memory leaks that would eat all the resources of the server. Also for the IT-pro’s that owned the SharePoint farms, it was a huge challenge to test all the code before deploying it. Because of this same reason, Microsoft did not allow you to deploy full WebParts to Office 365, but only SandBoxed WebParts.

The big change with the new app model of SharePoint 2013 is that now the apps do not run on the SharePoint server, but on their own (app) application server. This gives the developers lots of freedom to create all kinds of functionalities, without them having to worry whether or not what they build, was allowed to be deployed on the SharePoint farm.

But how about talking to core SharePoint information? Microsoft integrated the oAuth token based authentication model for Office 365 and the Server to Server (S2S) authentication model  for on premise. So with only a couple lines of code you have all the SharePoint methods you need at your disposal!

It all sounds like a great step forward to me, but my biggest concern is hosting the apps.
All the demos at SPC12 used Microsoft Azure to host their apps in combination with the new Office 365. Even though Microsoft says that it is very easy to host the apps yourselves and connect them to either Office 365 or your own on premise SharePoint 2013, this is something that we need to dive into before we can give our corporate clients good advice about migrating to SharePoint 2013 on premise or moving to the new Office 365!

SharePoint Conference 2012 Day #1

November 13th, 2012 Comments off

Monday was the first day of SharePoint Conference, and what a great day.

We will be doing some more in depth blogging, but for now this is my first impression.
Having only been to DevConnections before, I am amazed at how well Microsoft has organized this conference.
Many events can learn from Microsoft on how to manage a crowd of 10.000 people to enter and leave the Keynote area. And even more amazing is how they manage to feed that many people in an organized way. But that is not what we came for of course, we came for SharePoint 2013. And that also exceeds our expectations…

We are very busy working with EMC, as their C3P partner, specializing on integrating EMC Documentum with SharePoint for clients, and while we had already dived into SharePoint 2013 we did not really ‘get’ the huge changes. We get them now.

User experience

In the user experience area, which was good to begin with, SharePoint has made another huge leap. Expanding the MySite feature that has not really been used to its full potential, they have added SkyDrive Pro. This adds all documents users store in their personal space to the cloud, allowing access anytime, anywhere, from any device.
Basically, what users have used until now is Dropbox, SkyDrive or other cloud solutions. This is now added to SharePoint and has become the default storage location when creating documents with Office 2013.

On top of this there will be Apps for accessing all this from Windows Phone devices. But there will be one for iOS as well. This shows that Microsoft is very serious about wanting all users to be able to access their information anytime, anywhere.


The social capabilities of SharePoint have also been expanded to support even more collaboration, allowing users to really work together, learn from each other and use each others expertise. With the acquisition of Yammer and the work they have been doing together to integrate this into SharePoint, companies that already use both can get even more out of their social internal network.

Office integration

In the earlier office versions there were already possibilities to integrate with SharePoint but this has also been greatly improved. Combined with the new real drag and drop functionality users now really will not have to use the Office client they are working in all day. And with the addition of the App Model we can now add functionality to bring the users any SharePoint functionality they may need to do their daily jobs better and they can have this at their fingertips…

App model

This is another thing we did not get when we unboxed the SharePoint 2013 preview. We get it now…Contrary to what we could do with WebParts we can now build apps that support both the cloud version (Office365) as well as on premise environments. Apps run outside SharePoint but can have full access to all things SharePoint. Basically we can build Apps for anything. Supporting end-to-end solutions for clients but also adding magic to any of their Office clients. Definitely one of the areas we will be doing more blogging on.


Until now the limitations of Office365 and of sandboxed solutions have often resulted in us advising the client to stay on premise. Now with the new App Model we no longer have to and we can advise clients to take advantage of the new 90 day release cycle. Meaning that new cool features are available sooner instead of the 3 year server product release cycle. Which also will mean no more expensive upgrade cycles for clients. For us developers it means being able to use cool new features much sooner because we do not have to wait for clients to upgrade. For me that is a win-win situation on all sides.

We are now gearing up for our second day at SharePoint Conference. I for one cannot wait to dive more deeply into all of the above and into the areas we did not even cover yesterday.