CIT HEP Computing

This page is dedicated to centralize all computing related subjects

• ComputingGridTools
• ComputingT3HiggsFAQ

Workplan guidelines

Our priorities are in this order :

• Uptime
• User support (blocker issues)
• Resource utilization -- making sure that all hardware is being used
• User support (non-blocker important issues)
• Backups
• Monitoring
• User support (potential non-issues)
• Security
• R&D - This probably needs extra manpower. We have ideas but no time.

Extremely handy links :

MonaLisa HepSpec table

USCMS T2 HepSpec table

Sites pledges

Related links

• USLHCNET
• SC

Upgrades (software)

USCMS Upgrades twiki

Pledges, etc

• REBUS
• SiteDB

Monitoring links

We need a page to aggregate those, plus some DashBoard + PhEDEx + central CMS monitoring tools plots.

Central CMS

• Overview
  • Site Race
  • Batch system efficiency
  • Site Status Board
  • Site Readiness
  • HammerCloud

• Running production x Pledge
• Pledges and shares
• Grid jobs
  • Daily running
  • Daily pending
  • Weekly running
  • Weekly pending
  • Job efficiency (success/failure)
  • Production jobs efficiency (success/failure)
  • GWSSMon Caltech

• SAM Overview CIT_CMS_T2
• Nagios SE
• Nagios CE1
• Nagios CE2
• Xeon CE
• Caltech T2 links from T1s
  • Last 24 hours
• Transfers, from Caltech
  • Weekly rate
  • Weekly quality

• Transfers, to Caltech
  • Weekly Rate
  • 48h quality

• Perfsonar

• Analysis central monitoring per user

• Subir phedex page

• GlideIn monitoring page

• XrootD Monitoring

• XrootD DashBoard

• xrootd fallback

• FTS not-as-good monitoring

• FTS great new dashboard

OSG

• RSV probes
• Local RSV

Local pages/systems

• Internal batch system monitoring
• Tier-2 hadoop monitoring
• Tier-3 remote hadoop monitoring
• Nagios
• Ticket system
• Campus cacti
• MonaLISA Temperature
• Zabbix worker node checks
• Zabbix distribution of item for nodes

Documentation

• Chaging SITECONF settings
• CRAB3 mantra

Monitoring shifts

Daily

• Check Readiness
• Check Site Status Board
• Check PhEDEx Transfers (both directions)
• HDFS status, corrupted blocks
• Check Job Failure rate, reasons if there is a site problem
• Check Ganglia plots
• Check GridFTP Screen on Zabbix

Weekly

• Check RAID status on servers
  • SRM
  • CEs
  • GridFTPs
  • GUMS
    • Do this fast by : [root@t2-headnode-new lists]# pssh -i -h raid-check-list.txt "cat /proc/mdstat" | grep "blocks super" | grep -v chunks
  • PhEDEx
  • T3 Headnode
  • T3 Login node
  • T3 JBOD
    • Query with : storcli64 /c0/v0 show
  • Newman
    • Query with : areca_cli64 ; vsf info
  • LDAP2

• Check that the important servers still have a working backup
• Nodes / Cores / Storage counts - once we update the inventory and have defined totals.

Monitoring requirements

• RSV
  • Cert exists
  • Cert validity

• Nodes
  • GLEXEC
  • CVMFS
  • SWAP Trigger
  • IOPS on HDFS
  • IOPS on /
  • Proc / User
  • Open Files / User

• Servers
  • RAID states

• NameNode/HDFS
  • Health
  • Checkpoints
  • NN - All filesystems usage maximum alert

• External
  • PhEDEx data on transfers
  • Data from SAM
  • Data from RSV
  • Data from DashBoard - Blackhole nodes