Caltech

Started having consistent stage-out problems by Sep 11th. Throttled down the transfers to other sites so we can investigate the issue with less load on the GridFTPs.

Plots here

= Older notes = Found that there was a problem related to LHCONE peering. Their traffic from Caltech was going through CENIC, not LHCONE, which is sub-optimal. Purdue is following up with local network support.

Spreadsheet

UCSD

Observed ~450 MBps in distinct days, see monthly plot

Site notes

General status

Site notes

Caltech

Nebraska

GFTP issues

Error reason: TRANSFER globus_ftp_client: the server responded with an error 500 Command failed. : Unable to extend our file-backed buffers; aborting transfer.

Purdue

GFTP Issues :

Error reason: TRANSFER globus_ftp_client: the server responded with an error 500 Command failed. : Allocated all 1500 file-backed buffers on server cms-g004.rcac.purdue.edu; aborting transfer.

Florida

Vanderbilt

Plan for the exercise

As discussed in the meeting, we would like to use Caltech as the source site, as it managed to do 25/29 Gbps with its setup, being a good source for sites optimizing their configurations. Once everyone else optimizes their download configurations and we observe which rates we get to which sites, we could start rotating who is the source site, and see what are the maximum rates that we get from them. It is important that we have multiple sink sites, as even if there are limitations in sites, the others will add up to the total rate.

• Tuning PhEDEx download configurations, so LoadTest settings will correspond better to reality
• Observing how transfers behave at the FTS level, note if the Optimizer algorithm is a limiting factor or it actually helps to achieve the optimal setting of active transfers for the available bandwidth at a given moment.
• The logical limitations would have been removed, sites can focus on setting their upload rates as they want and optimize their GridFTP setups, start observing what are the best rates they can get out of the storage.

Participation of sites

In order to contact only the interested sites, please fill out the table below :

Site	Connectivity <-- -->	Participating	Notes
T2_US_Caltech	100G
T2_US_Florida	100G	N/A
T2_US_Purdue	100G	N/A
T2_BR_SPRACE	10G	N/A
T2_US_MIT	10G	N/A
T2_US_Nebraska	10G	N/A	Upgrading to 100G soon
T2_US_UCSD	10G	N/A
T2_US_Wisconsin	10G	N/A
T2_US_Vanderbilt	10G	N/A

• cmsfts3.fnal.gov
• fts3.cern.ch
• lcgfts3.gridpp.rl.ac.uk

In the long run, US Sites should use FNAL. But it might be worth to understand if other FTS servers have a different optimizer behavior and why.

this

However there is also this if you would like to read more.

### AGENT LABEL=download-debug-t2fts PROGRAM=Toolkit/Transfer/FileDownload DEFAULT=on -db ${PHEDEX_DBPARAM} -nodes ${PHEDEX_NODE} -delete ${PHEDEX_CONF}/FileDownloadDelete -validate ${PHEDEX_CONF}/FileDownloadVerify -accept '%T2_US%' -verbose -backend FTS -batch-files 20 -link-pending-files 300 -max-active-files 300 -protocols srmv2 -mapfile ${PHEDEX_FTS_MAP}

• Caltech - 100 Gbps

• UCSD - 10 Gbps

• Wisconsin - 10 Gbps
• Vanderbilt - 10 Gbps

USCMS T2 Transfers

This twiki is intended to aggregate all necessary information for the current effort of improving inter-T2 PhEDEx transfers in the context of USCMS.

It is known that the networks supported between the 7 sites are of high capacity and availability. However there seem to be some limitations to be addressed and tested at the level of CMS Transfer tools or configurations of these, that could improve the overall performance and at the end, make these systems perform better and deliver data faster between sites.

The general picture on transfers over 20 Gbps and some of these configuration problems are mentioned in Samir's talk at the T2 meeting of 07/29.

The ideal is that even 10 Gbps sites could participate, as it is possible that the currrent settings are not optimal for fast transfers. We could tune it until it saturates the 10 Gbps link and everyone would have exercised how to improve transfer rates in Debug.

Current connectivity at sites

So far, the showstopper was the uplink bandwidth for most sites. Since July 2014 this is starting to change, this is to track the current state from different sites

• MIT - 10 Gbps
• Nebraska - Could do more then 10 Gbps currently, will have 100 Gbps on the T2 soon.
• Purdue - 100 Gbps. Running tests with FNAL.
• Florida - 100 Gbps
• UCSD - 10 Gbps (Samir observed when running tests. To be confirmed by admins)
• SPRACE - 10 Gbps
• Wisconsin - 10 Gbps - Saturating from times to times. Would like to avoid tests for now.

FTS Notes

PhEDEx Documentation

PhEDEx configurations

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed condimentum rhoncus ligula, et lobortis ipsum aliquet id. In fringilla felis id venenatis placerat. Vivamus ornare egestas mattis. Donec sapien leo, gravida vitae magna sed, dictum convallis metus. Nulla mattis iaculis diam a gravida. Sed ac sem eget nisi tristique convallis vel sit amet dolor.

.
.
.
 -batch-files 500
 -link-pending-files 200
 -max-active-files 700
 -link-active-files 'T1_CH_CERN_Buffer=50'
 -link-active-files 'T1_DE_KIT_Buffer=10'
 -link-active-files 'T1_DE_KIT_Disk=10'
 -link-active-files 'T1_ES_PIC_Buffer=100'
 -link-active-files 'T2_RU_RRC_KI=2'
 -link-active-files 'T1_FR_CCIN2P3_Buffer=100'
 -link-active-files 'T1_FR_CCIN2P3_Disk=100'
.
.
.

-- Main.samir - 2014-07-29