Upload files to SharePoint using Office PnP PowerShell cmdlets

Imagine you have a folder like this:

and you would like to end up with something like this in SharePoint:

If you want to programmatically upload files to a SharePoint library, there are several options available:

1. CSOM custom code (Online and on-prem)
2. REST API's (Online and on-prem)
3. server-side PowerShell (on-prem)

Thanks to the work of the Office Dev PnP team there is at least one more option: The Office Dev PnP PowerShell cmdlets. And this option is beautifully simple, so I want to share it, especially since Google does not yet return any useful results showing this method.

In this post I will show you how to upload the contents of a folder to a SharePoint library using the PnP PowerShell cmdlets.

The actual cmdlet uploading the documents is Add-SPOFile (https://github.com/OfficeDev/PnP-PowerShell/blob/master/Documentation/AddSPOFile.md). The same cmdlets work with SharePoint Online and on-prem (2013 & 2016) versions. And if you want it to, the Add-SPOFile cmdlet can:

• Check-in uploaded files automatically (-Checkout parameter)
• Publish and approve (-Publish and -Approve parameters)

Prerequisites

First you will need the PnP PowerShell cmdlets installed. If you run Windows 10, just use the following PowerShell commands (select the version for your SharePoint environment)

• Install-Module SharePointPnPPowerShellOnline
• Install-Module SharePointPnPPowerShell2016
• Install-Module SharePointPnPPowerShell2013

For any other version of Windows you can download the release packages here: https://github.com/officedev/pnp-powershell/releases

The script

The following script will do the actual uploading magic:

That's it. The script will loop through all files in the specified folder and upload the files to a corresponding path in the SharePoint library. It will even create the folders in your library in the exact same hierarchy as you have on your local disk.

The good, the bad and the people picker

SharePoint 2016 was promised to make "SAML a first class citizen". The biggest problem with any SharePoint on-prem installation using something else than Windows authentication is the people picker, which does not provide means for searching for users in these scenarios. I'd like to see an out-of-the-box solution leveraging Azure AD, as most organizations are using Azure AD with Office 365 anyway. If you want to know my reasons, read on!

SharePoint 2016 and the first class citizen

I installed my first SharePoint 2016 farm yesterday. I went for SP 2016 Release Candidate (obviously), running on top of Win 2016 Technical Preview 4 and SQL Server 2016 Community Technology Preview 3.3. I was mostly happy with the installation experience, and although there were a few rough edges here and there, it went a lot more smoothly than I expected for a combo of three preview products.

I noticed some changes in the configuration process (MinRoles), but otherwise, everything looked very familiar. Much more familiar in fact than almost exactly 3 years ago with SharePoint 2013. I did not expect anything major though, so I was not disappointed.

There was however one feature I was eagerly looking forward to: SAML authentication (or Trusted identity provider authentication, as it is called in SharePoint, which is actually SAML-token based authentication using SAML 1.1 and WS-Federation, but don't let this confuse you too much).

Back at Ignite 2015, Bill Baer promised "SAML authentication becomes a first class citizen in SharePoint 2016" (don't take my word for it, he says it right at 11:30min on this video from Ignite: https://channel9.msdn.com/Events/Ignite/2015/BRK2188) and he goes on to say "we chose SAML authentication as a default". I was very excited to hear this.

I really like the concept of SAML authentication, and have implemented it in several real-world scenarios. The thing I like most about SAML, is the possibility to extend the claims ticket of a user with additional claims. You could, for example, add the users nationality to the claims ticket, and add a nationality claim (e.g. "German") to a user group in a SharePoint site. The beauty of this solution is, that SharePoint respects these claims, and will treat them just like traditional AD group based or user based permissions.

If you are not familiar with this concept, I highly recommend reading a series of blog posts on this topic by Steve Peschka (see the end of this post for links).

Is the people picker broken?

The problem? There is no out of the box possibility to do proper name resolution when using SAML authentication. I found a post titled "Is the people picker broken" when I was once again doing some research on this topic (http://www.sptechcon.com/news/is-the-people-picker-broken) and a suggestion on uservoice.com to "Ship SharePoint Server 2016 with a SAML Claims-aware People Picker" (https://sharepoint.uservoice.com/forums/329220-sharepoint-dev-platform/suggestions/8216679-ship-sharepoint-server-2016-with-a-saml-claims-awa).

Yes, it is broken.

You can, if you really want to, install a 3rd party claims provider (often but not quite correctly referred to as "people picker") or even write your own. This does sound promising, however, there is one major flaw with this.

The claims provider must inherit the SPClaimProvider class...

... and the SPClaimProvider class contains both claim augmentation and searching and resolving claims. This means, from a technical point of view, you are adding a component which will interfere with two processes in your SharePoint farm:

1. The log-in process (claims augmentation)
2. Selecting users (or claims) in the people picker (search and resolve)

Now I don't know about you, but most of the environments I've seen can go for a few hours or days without being able to use the people picker. Users will cry, and there will be angry emails, but everybody will be fine in the end. The same is not true for logging in. Most environments I've seen can definitely NOT go without the ability for users to authenticate for several hours.

When you install a claims provider, you are adding a single point of failure to your farm, which might break the whole authentication process, even though you only wanted to make it easier for your users to use the people picker. And we all know, authentication not working is a synonym for "EMERGENCY!!! Cannot access SharePoint!!!1". And I've seen this happening first hand.

If SAML is a first class citizen in SharePoint 2016...

... then I'd like to see the search and resolve functionality separated from the claims augmentation process. I really don't like the idea of having to mess with both, if I "only" want to provide search and resolve to users.

I have tried a few third party claim provider solutions, but I'm not happy with any of them. I also started implementing my own, but quickly gave up, once I noticed how badly I could break the authentication process.

My suggestion?

Most organizations I see use both SharePoint Online and SharePoint on-prem, in a "hybrid" scenario. The level of integration between the Online and on-prem flavors varies, but I'll call it hybrid anyway. For this case, why not provide the possibility to do people picking against the Azure AD directory the organization is using with O365? People picking might get a consistent experience throughout Online and on-prem sites, and a very large portion of cases would be covered. Do this, and I'll believe SAML is a first class citizen in SharePoint 2016.

Further reading on the SAML with SharePoint topic:

Steve Peschka on SAML and custom claims providers:
Part 1: https://samlman.wordpress.com/2015/02/28/writing-a-custom-claims-provider-for-sharepoint-2010-part-1-claims-augmentation-and-registering-your-provider/
Part 2: https://samlman.wordpress.com/2015/02/28/writing-a-custom-claims-provider-for-sharepoint-2010-part-2-adding-support-for-hierarchy-nodes/
Part 3: https://samlman.wordpress.com/2015/02/28/writing-a-custom-claims-provider-for-sharepoint-2010-part-3-searching-claims/
Part 4: https://samlman.wordpress.com/2015/02/28/writing-a-custom-claims-provider-for-sharepoint-2010-part-4-supporting-resolve-name/

Kirk Evans on fixing the people picker:
https://blogs.msdn.microsoft.com/kaevans/2013/05/26/fixing-people-picker-for-saml-claims-users-using-ldap/

We couldn't complete the updates...

I was working on updating my Windows Server 2012 R2 VM running in Azure with the latest Windows Updates (as of January 11, 2016). Installing updates failed over and over again with a "We couldn't complete the updates, Undoing changes" message after reboot. This affected all updates I was trying to install in one "batch".

I googled, and found an instruction to disable Secure Boot in Hyper-V (http://blog.powerbiz.net.au/fixes/we-couldnt-complete-the-updates-undoing-changes-error-on-windows-server-2012-r2/) but I cannot do that in Azure. But I noticed I could install updates one by one, so I did that until I found the problematic update.

In my case, the root cause seems to be update KB3096411 (https://support.microsoft.com/en-us/kb/3096411). After excluding this one, all other updates installed fine.

Update January 12, 2016:
The problematic update KB3096411 fixes some issue with KB3000850 (https://support.microsoft.com/en-us/kb/3000850), which had problems with installation on Windows Server instances running on a platform with Secure Boot enabled (UEFI hardware or 2nd generation Hyper-V), giving the same error code I am getting (0x800f0922). The solution recommended by Microsoft was to install update KB2975061 (https://support.microsoft.com/en-us/kb/2975061), which I have now done, but the problem remains. KB3000850 installed fine, but KB3096411 fails.

Azure Premium Storage

I'll give you some background on this topic. I work with SharePoint. A lot. And although a lot of clients are moving to SharePoint Online for whatever they are doing with SharePoint, we still work with numerous on-prem environments.

For developing, testing and troubleshooting against on-prem SharePoint it has become common practice to run Windows Server virtual environments with SharePoint running on them on your own laptop. Now, if you have ever looked at the system requirements of SharePoint 2013 (and if you have read this far then you probably have) then you know that SharePoint is a resource hog. The obvious solution 3 years ago when SP 2013 was published, was to get ridiculously powerful laptops to get your work done. You know, the kind you can use to heat your apartment, with a power brick the size of a, well, actual brick.

After three years of running around with this beast of a laptop (which even after three years is still really fast), I want to get rid of this burden, and try to find some other solution.

Running SharePoint on Azure virtual machines

There are a lot of articles on how to move a Hyper-V virtual machine to Azure for running it in the cloud, so I won't go into detail on that. Check out this article if you want to know more: https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-create-upload-vhd-windows-server/. But DO NOT SYSPREP your SharePoint VM. In my tests, it did not survive that.

The problem? Azure Storage is slow. Really really slow. It is reasonably fast when reading and writing large, sequential pieces of data, and once your server has loaded DLLs and whatnot into RAM, your machine will be fast. For random reads and writes of small bits of data it is horribly slow though. For databases you can set up striped disks and get good IOPS from standard storage, but then... I don't want to set up development environments with striped disks for SQL databases and my Visual Studio installation and then have to copy an array of .vhd files around every time I need a new copy of the environment. Don't get me wrong, Azure is great for running VMs, but running a single server SharePoint dev environment with just one disk is a real pain.

I want a single .vhd, containing Windows Server, SharePoint, Visual Studio and whatever I want to use for development. And I want things to happen reasonably fast. Luckily, Microsoft has introduced Premium Storage, SSD backed storage with the only purpose of providing better performance for IOPS heavy workloads in virtual machines (and me playing around with SharePoint and Visual Studio is IOPS heavy).

If you have used Azure for anything recently, you might know that:

1. Microsoft is replacing the old management UI (https://manage.windowsazure.com/) with a new portal (https://portal.azure.com/)
2. Microsoft is moving from the "Classic" deployment model to a new "Resource Manager" deployment model

This means Azure is currently (as of January 7, 2016) a pretty confusing mix of things, and I had a hard time figuring out some things when trying to get my virtual machine into Premium Storage. So I thought this might be an ideal topic for a first post in my new blog.

What is Premium Storage

First of all, you can only use Premium Storage for disks. All other Storage services are backed by Standard Storage. There are currently 3 tiers in Premium Storage. The tier (if I understood correctly) is determined only by the size of the .vhd you put in there. So e.g. if your .vhd is 130 GB, it will be in the P20 tier. Check the Azure website for current pricing on Premium storage (https://azure.microsoft.com/en-us/pricing/details/storage/).

These are the Premium Storage tiers and their respective performance promises:

Tier	P10	P20	P30	Standard storage
Disk size	128 GB	512 GB	1024 GB	max 1023 GB
IOPS per disk	500	2300	5000	max 500
Thoughput per disk	100 MB/s	150 MB/s	200 MB/s	??? MB/s

Now this is confusing, isn't it? The P10 tier offers the same 500 IOPS as a Standard Storage disk does. Or so you'd think. When you use Premium Storage, according to Microsoft, Azure "provisions for you a guaranteed number of IOPS as per the disk specification". I did not find any promise for Standard Storage. So lets find out if there is a difference.

Premium Storage can only be used with DS and GS Azure VMs. The pricing for DS and GS series is the same as for the equivalent D and G series. There are two main differences between the S and non-S machines:

1. *S-tiers are enabled for Premium Storage
2. *S-tiers allocate a large part of the local SSD temp storage in D and G series as cache, and leave twice the amount of RAM as temp storage (e.g. D2 has 7 GB of RAM and 100 GB of SSD temp storage, whereas DS2 has only 14 GB of SSD temp storage and uses the remaining 86 for caching of the persistent disks)

As you were probably not using the temp disk with your SharePoint VM anyway, this is not going to be a problem. (Or maybe you want to live a little, put your content databases there, or even better, your work in progress source code?)

How to set up a VM with premium storage

For the purpose of this post, I'll use the classic deployment model. I will have to learn the resource manager model, that's another thing I've not come around to do.

I mostly follow this guide for setting up my Premium VM: https://azure.microsoft.com/en-us/documentation/articles/storage-migration-to-premium-storage/

Before you start, I assume you have the following ready:

1. Azure subscription (obviously)
2. .vhd containing your VM in Azure Storage (the "Standard" kind)

Next we will:

1. Create a Premium Storage account (and storage container)
2. Copy a .vhd to the Premium Storage account
3. Fire up a DS virtual machine from the existing .vhd
4. Compare performance with a D series VM

Creating a Premium Storage account is fairly easy. You can do it either using PowerShell or using the new Azure Portal. The old management UI does not allow you to create Premium Storage accounts.

Creating a Premium Storage account

To create a Premium Storage account in the Azure Portal, navigate to Storage accounts (I used the classic kind), select Add, give your storage account a name and select the Premium-LRS pricing tier. Easy. Then create a storage container and call it "vhds".

Alternatively, you can use Powershell, in my case the commands look like this:

New-AzureStorageAccount -StorageAccountName "yourstorageaccountname" -Type "Premium_LRS" -Location "North Europe"

Set-AzureSubscription -SubscriptionName "Your Subscription Name" -CurrentStorageAccountName "yourstorageaccountname"

New-AzureStorageContainer vhds -Permission Off

Make sure you have a fairly new version of Azure Powershell installed for this (I'm currently using 1.0.2, as of January 8, 2016). According to MSDN you can set the Resource Group when creating the storage account using PowerShell, but that did not work for me, so I set the Location to "North Europe" manually.

Copying VHDs to Premium Storage

Now you should use Powershell. Microsoft tells you, that you can also use AzCopy or Azure Storage Explorer, but I find Powershell to be the most effective and reliable by far. (Especially uploading large files to Azure Storage works much faster and is more reliable with Powershell than any other tool I've tried. But this is almost a topic for another post.)

Copy your .vhd to the newly created Premium Storage account like this (make sure to set source and destination account names and keys according to your environment):

$destContext = New-AzureStorageContext -StorageAccountName $destAccountName -StorageAccountKey $destAccountKey

$srcContext = New-AzureStorageContext -StorageAccountName $sourceAccountName -StorageAccountKey $sourceAccessKey

Start-AzureStorageBlobCopy -SrcBlob "YourVHD.vhd" -SrcContainer "vhds" -DestContainer "vhds" -Context $srcContext -DestContext $destContext

Creating a VM from the uploaded .vhd

For this step you can again use either Powershell or the Azure Portal and Powershell .

First, you need to "register" it as a Disk which can be attached to a VM. I did not find this operation in the Portal, so use Powershell:

$vhdLocation = "https://yourstorageaccountname.blob.core.windows.net/vhds/YourVHD.vhd"
Add-AzureDisk -DiskName "YourVHD" -MediaLocation $vhdLocation -Label "My SharePoint 2013 VM" -OS "Windows"

Now you can create a VM from this disk. If you use Powershell, make sure you set the InstanceSize to a DS series such as "Standard_DS2". If you have created VMs from disks in the old management UI, then you will have a hard time doing the same in the new Portal. Before, you went to create a new VM under Virtual Machines, and then selected the disk you wanted to create from. Now in the new Portal, you go to Disks, and under your disk, you'll find a button to create a new VM.

Now wait a few minutes, and you are ready to connect to you Premium VM! Now lets see how it compares to a Standard VM.

Performance comparison

For the sake of this blog post, I created two VMs from the same .vhd (from two separate copies obviously). A DS12 with Premium Storage and a D12 (or rap-machine, as my colleague likes to call them) with Standard Storage. For the sake of possibly saving some money, I also fired up a DS11 VM (which costs half as much as a D12), to see how it performs compared to a D12 VM.

First I measured disk speed using CrystalDiskMark. The performance difference is remarkable. For sequential reads the Standard Storage disk looses, but the real difference come with random, small reads and writes. For random 4kb reads, Premium Storage performed 30 times better than Standard Storage. P10 with a DS11 VM gets about half of the DS12 results shown below, because the storage bandwidth is throttled to 64 MB/s.

To see if there is any real-world performance benefit, I tried a few real-world tasks someone working with SharePoint will be facing often. Manual tasks performed through UI were measured a few times (by hand) and the result is an average. To measure the duration of creating site collections in Powershell, I used [System.Diagnostics.Stopwatch] (very handy by the way).

Task	D12	DS12	DS11
Creating a site collection from Central Admin	1 min 59 s	33 s	35 s
Creating 10 site collections using Powershell	16 min 56 s	4 min 56 s	5 min 18 s
Opening a team site after running IISRESET	1 min 27 s	50 s	60 s

That is a clear performance boost! Creating site collections takes 4 times longer on a Standard D12 VM than it does on a DS11 or DS12 machine. Everything else I tried, like installing Windows Updates, installing SharePoint CUs, using Visual Studio, felt a lot snappier than on the D12 VM.

Lessons learned

1. 500 Premium IOPS is more than 500 Standard IOPS.
2. If you are setting up a "single server" SharePoint development VM, use Premium Storage. It's probably worth it.
3. DS11 with a P10 disk is in most cases probably the best "bang for your buck" solution.
4. Don't even think about setting up a SharePoint VM using Basic Tier VMs. You'll grow old before you get anything done.
5. I'm starting to get used to the new Azure Portal.

To-do list

1. Try the P20 and P30 Premium Storage tiers
2. Learn the resource manager model

Welcome and Happy New Year

I've been toying with the idea of starting a blog for quite some time but I've just never come around to actually do it. Now the new year seemed like a reasonably good time to actually start writing, so here it goes.

The next problem was finding a title for this blog. I'm planning to write mostly on topics related to IT, so I wanted to include that in the title. Additionally I'm part German and live and work in Finland, so why not include that as well.

A while back this picture was floating around on the internet:

Give credit where credit is due

I first thought this was just another meme, but I did my research. This picture of four paper clips with the caption Chaos. German style. is actually part of an ad campaign for Bank Forum, a Ukrainian subsidiary of the German Commerzbank. You can check out the whole campaign here or here.

There is also a short video on the homepage of the Hungarian copywriter, Karolina Galácz, who was behind the campaign back in 2010 (check it out here: http://karolinagalacz.com/The-German-Style-Campaign). And last but not least, the TV-ads were directed and edited by Finns (Mikko Lehtinen and Miika Vaso). So this is kind of a perfect match for me.

Please enjoy:
Finnish IT. German style.