import mdf_toolbox

Globus Transfer Utilities

While using the Globus Web App to perform data transfers is easy, programmatically doing the same has some overhead, which these utilities try to reduce.

quick_transfer

quick_transfer() starts a transfer and monitors its progress. You must have a TransferClient already created and authenticated (see the Authentication Utilities tutorial).

You must provide:

transfer_client: An authenticated Transfer client

source_ep: The source endpoint ID

dest_ep: The destination endpoint ID

path_list: A list of tuples, where the first element is a source path, and the second element is the destination path. Directory paths must end in a slash, and file paths must not.

transfer_client = mdf_toolbox.login(services=["transfer"])["transfer"]
source_ep = ""
dest_ep = ""
path_list = [
    ("/home/source/files/file.dat", "/home/destination/doc.dat"),
    ("/home/source/all_reports/", "/home/destination/docs/reports/")
]

mdf_toolbox.quick_transfer(transfer_client, source_ep, dest_ep, path_list)

It is normal for Globus Transfer to encounter a few transient errors when transferring data. The transfer is very robust and can usually recover from most issues. By default, quick_transfer() will tolerate 10 error events before cancelling the transfer. You can customize this with retries, with -1 indicating infinite retries (until the transfer times out naturally).

Additionally, you can customize the poll interval (in seconds) with interval.

mdf_toolbox.quick_transfer(transfer_client, source_ep, dest_ep, path_list,
                           interval=7, retries=-1)

custom_transfer

custom_transfer() is similar to quick_transfer(), but allows you greater control over the finer details. It is also more complex.

All of the required parameters from quick_transfer() are required for custom_transfer(). interval is also available. However, you can additionally set the natural timeout due to inactivity (in seconds) with inactivity_time.

The major difference between the two transfer utilities is that custom_transfer() will yield any error events and allow you to .send(False) to cancel the transfer.

transfer_generator = mdf_toolbox.quick_transfer(
                                    transfer_client, source_ep, dest_ep,
                                    path_list, inactivity_time=600)
res = next(transfer)
print(res)
try:
    while True:
        res = transfer.send(True)
        print(res)
except StopIteration:
    pass

get_local_ep

get_local_ep() attempts to discover the local machine’s Globus Connect Personal endpoint ID. It does not properly detect Globus Connect Server endpoints, and can sometimes return inaccurate results if the local machine is not running GCP. If there are multiple possible endpoints, the user is prompted to choose the correct one.

get_local_ep() requires a TransferClient.

If possible, it is recommended to get the local EP ID some other way, such as through the Globus Web App or by asking the endpoint’s owner.

mdf_toolbox.get_local_ep(transfer_client)