1. Home
  2. Jobs
  3. Advanced Fields – Source and Target Nodes

Advanced Fields – Source and Target Nodes

Advanced fields are settings for unusual use cases. Other than the fields for Exclude and Include Files in the Windows Drive or Share System, advanced fields may be left at the default values unless you are directed otherwise by Shuttle Tech Support.  Some fields listed below will only be available for some systems and not for others.

Advanced Fields

Enable Async Metadata Read This feature enables Shuttle to list the entire contents of an element selected in your Microsoft (OneDrive or SPO) source or target node in a single rest call. It makes the listing of Microsoft sites and directories much faster and more efficient under most circumstances, but is optimized for large folders and, in data sets that have many small folders, performance may be better without AMR. Default for this option is checked on; leave it checked unless directed to do otherwise by Shuttle support.

Exclude Files  Enter a text string in conjunction with wildcard characters (*) to exclude files from transfer.  For example, if you do not want to send .docx files, enter *.docx in this field.  This field can be useful to exclude files that are generating permissions errors, either because the files are open by an application or because they have user-specific restricted access.  You must enter each excluded file type in a separate instance of the Exclude Files field. To create a new instance of the field, click on the (+) sign to the right of the field. The default Exclude Field excludes all .tmp files and files starting with the characters ~$ (which is also a temp filename notation).  If you want to exclude a folder, it is recommended that you only enter the folder name, without a preceding slash.  Note that exclude filters do not apply to the top level folders of the selected data set.  So if you have selected a folder /folderA, and you apply an exclude for ‘folder,’ /folderA will be transferred, but a subfolder /folderA/folderB will be excluded from transfer.

The above methods are for globs, which is the easiest implementation for exclusions. Shuttle also supports regex exclusions, in which case you should start the string with a “^” to signify the start of the regex string, and a “$” to signify the end of it. For example, ^(.*)?\.txt$ would exclude all txt files.

Force Shared Folder Depth For Dropbox target jobs. Allows a user to specify a folder depth where if a folder is created within that depth inside a matching team folder name, then that newly created folder becomes a shared folder. Leave this option set at the default of 0 unless otherwise specified by Shuttle support.

INCLUDE FILE METADATA  If this checkbox is selected, Shuttle will include semi-proprietary cloud file metadata such as descriptors in the transfer. While the metadata may be transferred, it may not be readable on the target platform.

Include Files  Enter a text string in conjunction with wildcard characters to limit a transfer to only the files specified.  For example, if you enter sendme.* in this field, all files that have a filename whose root name is ‘sendme’ will be sent, but any other files will be ignored.

The above methods are for globs, which is the easiest implementation for exclusions. Shuttle also supports regex inclusions, in which case you should start the string with a “^” to signify the start of the regex string, and a “$” to signify the end of it. For example, ^(.*)?\.txt$ would exclude all txt files.

INCLUDE SHARED FOLDERS  Includes folders in the file chooser that have been shared with you, but that are owned by another user in your corporate cloud account.  You must select this field on Dropbox target nodes if you are migrating to Team Folders.

IMPORTANT:  Do not select more than one account for transfer if Include Shared Folders is selected, as this can result in transferring an instance of the same folder for the owner and every sharee of the folder. That causes duplicate data on the target and excess byte charges for transferring the duplicate folders.

KEEP PARENT PERMISSIONS  If this box is selected, parent permissions on folders will be inherited by child items, even if the child item has reduced permissions. Note that this strategy can be useful if you are trying to minimize the number of ACLs that are applied to a given Microsoft subsite on OneDrive or Sharepoint Online.

Keep Existing Permissions Do not delete any permissions on the target when applying ACLs.

Limit Recursion Depth  This field will stop scanning the source data at the folder depth specified on account map runs only, and any deeper folders will be completely ignored in the map generation. It will not affect simulation or transfer runs. If you just want to report the shares are at the top levels of your data set and not list out shares in the mapping sheet that are ten levels down, this is a useful feature that will also shorten the overall scan time for your map generation. Enter a value of ‘1’ to only scan the top level of folders if you have selected the root of one account, or have selected several accounts in the job configuration.

LIST ALL PERMISSIONS  This field stores the permissions associated with every item – file and folder – in the job. Depending on the characteristics of the source and target, and how permissions are shared, the listings in the spreadsheet will be significantly less than this. For example, inherited permissions will not be listed unless there is something that either breaks or expands upon that inheritance. Default for this option is selected; Shuttle recommends that you leave this option set at the default.

MAX CONCURRENT FILES DOWNLOAD  An index of how many files Shuttle will deliver concurrently to the target. Shuttle will optimize this field for you; you may leave it at its current setting unless instructed otherwise by Shuttle.

MAX CONCURRENT FILES UPLOAD  An index of how many files Shuttle will send concurrently from the source. Shuttle will optimize this field for you; you may leave it at its current setting unless instructed otherwise by Shuttle.

MODIFIED BEFORE / MODIFIED AFTER  If you enter a date in the Modified Before field, Shuttle will only transfer files modified on or before this date.  If you enter a date in the Modified After field, CloudfastPath will only transfer files modified on or after this date.

This feature will filter out all folders from transfer that are ultimately empty when their child items are filtered out by the Modified filter. It will filter out empty folders regardless of whether the “Do not send empty folders” option is selected or not.

Do not send empty folders If this option is selected, empty folders will be listed as filtered in the transfer report and will not transfer. If there is a string of empty folders – e.g., /empty folder1/emptyfolder2/emptyfolder3 – the full tree will be filtered.

SELECT ALL SITES (SHAREPOINT)  Select this checkbox to move all SharePoint teamsites.

SHAREPOINT DOMAIN  This parameter is used to perform the NTLM handshake. Most SharePoint installations should leave this field blank.

SHAREPOINT WORKSTATION  Similar to Sharepoint Domain; used only for NTLM handshake. Most SharePoint installations should leave this field blank.

SHAREPOINT TEAM SITE  This is where you select the team site for the host named site collection that was specified for the ‘Hostname’ on the system. This is the only way to select the team site. Once you have all the other fields on the node and system specified, except for Source Folder, you should enumerate the team sites using the refresh button, and select which one you want to migrate.

Suppress Email Notification for Sharing This setting, for jobs that have OneDrive or SPO as a target, suppresses email notifications for when items are shared with unregistered external (guest) sharees. Email notifications for registered guest sharees and internal sharees are suppressed independent of whether this option is set or not. Note that an email to an unregistered guest sharee does two things: it notifies the user that s/he has data that is shared, and provides a link in the email to register with the tenant as a guest user. If you suppress these emails, the user can be registered manually as a guest, but it requires admin intervention. Once the external sharee registers as a guest and is known to the tenant with a user id, they will not receive further sharing notifications.

Suppress Email Notification for Collaborations  This setting, for jobs that have Box as a target, suppresses email notifications to all internal and external sharees for folders and files when selected. If it is not selected, sharees will receive an email notification for every target item that is shared with them. Regardless of whether this setting is selected or not, file and folder owners will still receive notification for all of their shared items. The email subject takes the format “[Box Username] has accepted the invitation to your [‘folder or file name’] [folder|file] on Box”, so advising users to set up email filters for the beginning string of the email can mitigate mailbox clutter during migrations to Box.

SYNC FILE VERSIONS  Select this option if you want to migrate versions of files in the selected data set. Note that file versioning is only supported for newly migrated files to the target; you cannot transfer previous versions of a file that already exists on the target. A given job should have versioning either set or not set for all instances of that job; do not change the setting mid migration.

If versioning is selected for a job, Shuttle will transfer the mtimes for all versions and save them with the file metadata. However, the Box web app will display the server-side (i.e., time uploaded to Box) mtime for versions of the file, and those times will be labelled as “Uploaded” stats. You can, however, display the client-side (native) mtime for the file by going to Properties > File Information, which will display the client side mtime for the top version (which will be the most recent version unless the version order is edited).

MAX FILE VERSIONS  This field displays if SYNC FILE VERSIONS is selected. Default for this field is 0, which will transfer all available versions, up to what is supported by the target service provider.

Version limits for Box are:

  • Business/Business Plus: 50 versions
  • Enterprise:  100 versions

Check the settings in your Box account prior to migration if you anticipate transferring a lot of versions, as max versions is configurable and may be set to a lower value than the max supported.  Consult your Box service representative if you have any questions.

For all source adapters except Google Drive: Enter the max number of versions of files that you wish to transfer, in addition to the current version. Example:  if you enter 2 in this field, Shuttle will transfer the current version of this file plus the two most recent versions (if that many are available), so a max total of three versions will transfer.

For Google Drive source adapter: Enter the max number of versions of files that you wish to transfer, including the current version.  Example:  if you enter 2 in this field, Shuttle will transfer the current version of this file plus the most most recent version, so a max total of two versions will transfer.

USE LAST MODIFIER AS UPLOADER  This option will write the last uploader from the source to the Last modifier metadata on the target.  Source last modifiers are mapped to Target last modifiers to accomplish this, so this option requires Account Mapping; it is selected by default for account mapping jobs.  This can result in several additional source users being listed in the Users tab of the spreadsheet and described as “updaters” or “creators” in the Information column. Note that if a Last Modifier user on the source has been deleted, they will be listed in the account mapping spreadsheet as a “Prior Collaborator.”  Prior Collaborators are best mapped to a target admin, in which case they will display as being last modified by the admin.  This feature applies to files only on most platforms.

If the last modifier for a given file is not available for any reason – for example, if a creator or updater is skipped in the mapping spreadsheet – the target owner will be assigned as the last modifier for that file.

Additional details for how last modifiers are handled for Box targets are available on the Box Admin Adapter page.

Content Scanning  If this option is selected, it will scan Microsoft Office files and report out links embedded in these documents for other files. The locations of linked files are reported out in the simulation and transfer reports (See About Linked Files for additional information). This function may increase scan times and Win app resources required on jobs. Shuttle will only scan Office files for links, so if the data set does not contain Office files, turning Content Scanning off will not improve job performance.

Some examples of links reported include:

  • links to other spreadsheets used in VLOOKUPs and other functions in Excel files
  • actual links embedded in the text
  • embedded content from other files, such as Excel charts from one file that are embedded in a second file
  • templates (.dotx files) for Word documents

Migrate Box Notes in Native Format  If this option is selected, Box Notes will be exported in native JSON format. If this option is not selected, Box Notes will be exported to .docx files. Default is unselected, meaning Box Notes will be exported to .docx files.

WINDOWS DRIVE OR SHARE ADVANCED FIELDS  Windows Drive or Share systems have several advanced fields that enable you to transfer files that are not transferred by default.

Benchmark Mode If this field is selected, when the job runs, Shuttle will generate 100 test data files on the source in a folder called ‘cfptest,’ which will be saved as a subdirectory in the source folder as selected by the file chooser. Aside from the top level folder selected in the file chooser, any additional files and folders selected will be ignored for transfer. The Shuttle-generated test files will be sent to the target, and when the job completes, the Shuttle-generated files and ‘cfptest’ folder will be deleted. This field is intended for performance testing.

Generic Folder Listing This field is used for drives such as Linux that may have duplicate folder and file names.

Copy Hidden Files this will copy hidden Windows system files such as .sys and .ini files. By default, Shuttle does not transfer these files as they are generally not of use beyond the source computer.  Use this option and the Copy System Files option with caution, particularly if these files may eventually be downloaded to another Windows computer.

Default Account Name Enter a default name that should appear, for mapping and reporting purposes, as the owner of folders where the true owner cannot be accessed by the Windows app’s installed user. For example, you may enter ‘Shuttle User’ or any other string in this field; it does not have to be a real user. ‘Shuttle User’ will then be designated as the owner of any folder that the Windows app cannot determine the owner due to permissions, and will be listed as a source owner in the users tab of the spreadsheet. You will need to map the ‘Shuttle User’ as you would any other user, and ‘Shuttle User’ will appear as the source owner in all analysis, simulation and transfer reporting. This field must be completed before generating the map.

Dereference Links  This option is selected by default. When selected, Shuttle will follow the path of a symlink/shortcut to the file(s) it points to, and will transfer those files.

Do Not Scan Excluded Folders  If a folder is listed as an Excluded item, it will not be scanned. This can significantly speed up migrations if a large folder is excluded. Folders that are skipped in the mapping sheet are not scanned except for children that are unskipped, so this option will have no effect (or improved perfornance) for skipped folders.

Domain Host  This field is used for Account Mapping only. It specifies a server within the domain to which file owners belong, and may be used in cases where the computer on which the source files reside is outside of the domain for the file owners.  You may leave this field blank unless instructed to complete it by Shuttle Support.

Copy System Files  this is another option that enables you to transfer Windows system files.

Zip Special MacOS Files on NTFS  checking this option will zip the data and resource fork files for legacy MacOS font s into zip files that are more readily transferrable and stable on non-Mac platforms such as cloud providers.  Consult shuttle-support@box.com if you are interested in using this option.

Updated on September 22, 2021

Related Articles