Tutorial: Apache Nifi upload file to SharePoint with template

Explain how to connect apache Nifi to SharePoint API

6 min readFeb 25, 2023

This tutorial will provide a step-by-step tutorial about uploading one or multiple files to SharePoint using Apache nifi. Here, I also supply together the actual Apache nifi template.

SharePoint App Registration

Go to your SharePoint site, and register an app on the next site by changing <YourSiteDomain> and <NameOfYourSite> in the URL.

Syntax:


https://<YourSiteDomain>.sharepoint.com/sites/<NameOfYourSite>/_layouts/15/appregnew.aspx

Example:

https://contoso.sharepoint.com/sites/ContosoTeamSite/_layouts/15/appregnew.aspx

On App information page, click the respective Generate button, for Client Id and Client Secret.
Title: Nifi to Sharepoint
App domain: localhost
Redirect URI: https://localhost
Finally click in Create button

Note down the Client Id and Client Secret values generated. We will be using them in the next steps.

SharePoint grants Permissions to Add-In

Now, we have to set permissions and scope for that add-in and navigate to the next site by changing <YourSiteDomain> and <NameOfYourSite> in the URL.

Syntax:

https://<YourSiteDomain>.sharepoint.com/<NameOfYourSite>/_layouts/15/appinv.aspx

Example:

 https://contoso.sharepoint.com/sites/ContosoTeamSite/_layouts/15/appinv.aspx

Paste your Client Id in “App Id” (Generate previously) box and click on the lookup button.

After that, in the “Permission Request XML” enter the XML code below and click on the “Create” button.

<AppPermissionRequests AllowAppOnlyPolicy="true">
<AppPermissionRequest Scope="http://sharepoint/content/sitecollection/web" Right="FullControl" />
</AppPermissionRequests>

This will redirect to your page, where we have to trust the add-in to have the required permissions.

Get the Tenant Id

We need Tenant Id also. So to get the Tenant Id navigate to the next site by changing <YourSiteDomain> and <NameOfYourSite> in the URL.

Syntax:

https://<YourSiteDomain>.sharepoint.com/<NameOfYourSite>/_layouts/15/appprincipals.aspx

Example:

https://contoso.sharepoint.com/sites/ContosoTeamSite/_layouts/15/appprincipals.aspx

The value after “@” is your Tenant ID, Note down the Tenant ID, We will be using them in the next steps.

That’s it, app permission registration is done!

Let’s go to the Apache Nifi dataflow

This is how the process looks like.

List files

Processor: (ListFile) We are looking for get the list of files to upload, filling next statements:

Input Directory: Is the path of your folder, who contains the files to upload.
Recursive Subdirectories: True, I want to search all files in this path.
Record writer: we need to extract a list of files in CSV format, using CSV Record Writer.
File filter: default value is: [^\.].* that means types of files will upload, but, in my case, I only want .csv files, for this reason, I use .*\.csv.

This Processor generates a CSV with all files in the folder.

2. Put the List files to attributes

Processor: (PartitionRecord), thanks to that we will be able to do two things, first split each row in one file, and two, set values of filename and path in the attributes. These two things will help us later.

Record reader: Set CSV reader
Record writer: Set CSVrecord writer

After that, Set a par of values one for the filename and the other for the path how in the last image, by clicking on the plus at the top.

3. Set Parameters

Processor: (UpdateAttribute), Here we put all parameters who want to use in the process.


+----------------------+------------------------------------------+
|       Property       |                  Value                   |
+----------------------+------------------------------------------+
| client_id            | Client Id of the app created earlie      |
| client_secret        | Client Secret of the app created earlier |
| grant_type           | client_credentials                       |
| name_site            | ContosoTeamSite                          |
| SharepointDomainName | <YourSiteDomain>.sharepoint.com          |
| tenant_id            | tenant_id of the app created earlier     |
+----------------------+------------------------------------------+

Example:

4. Attributes to Body

Processor: (ReplaceText), Here we put the request in the body of flow file setting this property:

Replacement Value: enter the code below which is the body of the request with all parameters set

grant_type=${grant_type}&client_id=${client_id}@${tenant_id}&client_secret=${client_secret}&resource=00000003-0000-0ff1-ce00-000000000000/${SharepointDomainName}@${tenant_id}

Now, we have one flow file for each file in the list that we extracted in the first point (list files csv).

and the flow file looks like:

We already did the first part, now we go with the one on the right

Get token Sharepoint

Processor: (InvokeHTTP), We are now ready to make the request for the token to SharePoint.

HTTP Method: POST
HTTP URL:

https://accounts.accesscontrol.windows.net/${tenant_id}/tokens/OAuth/2

Request Content-Type: application/x-www-form-urlencoded
Response Generation Required: true

Extract token Sharepoint

Now we need to extract acces_token from the flowfile, using EvaluateJsonPath Processor

Setting:

access_token: $.access_token

Fetch File

Processor: (FetchFile), Now that we have to Fetch all files that we want to upload to SharePoint.

Adjusting the following property:

File to fetch:

${path}/${filename}

Upload Files to Sharepoint

Finally, we want to upload all files to SharePoint using apache nifi and processor: (InvokeHTTP), for that we need to adjust the following properties:

HTTP Method: POST
HTTP URL:

https://${SharepointDomainName}/sites/${name_site}/_api/web/GetFolderByServerRelativeURL('/sites/${name_site}/Shared Documents')/Files/add(url='${filename}',overwrite=true)

In my case, I want to upload all files to the main folder “Shared Documents”, here you can adjust the path of SharePoint folder destination.