• • GWSSMon Caltech

• Xeon CE

• Check PhEDEx Transfers (both directions)

• Check Job Failure rate, reasons if there is a site problem
• Check Ganglia plots
• Check GridFTP Screen on Zabbix

• • • Do this fast by : [root@t2-headnode-new lists]# pssh -i -h raid-check-list.txt "cat /proc/mdstat" | grep "blocks super" | grep -v chunks

• • • Query with : storcli64 /c0/v0 show
  • Newman
    • Query with : areca_cli64 ; vsf info
  • LDAP2

Monitoring shifts

Daily

• Check Readiness
• Check Site Status Board
• Check PhEDEx Transfers
• HDFS status, corrupted blocks
• Check Job Failure rate, reasons if there is a site problem
• Check Ganglia plots
• Check GridFTP Screen on Zabbix

Weekly

• Check RAID status on servers
  • SRM
  • CEs
  • GridFTPs
  • GUMS
  • PhEDEx
  • T3 Headnode
  • T3 Login node
  • T3 JBOD
• Check that the important servers still have a working backup
• Nodes / Cores / Storage counts - once we update the inventory and have defined totals.

• • Site Race

• Caltech T2 links from T1s
  • Last 24 hours

• xrootd fallback

• SAM Overview CIT_CMS_T2

• • HammerCloud

• • HammerCloud

• Nagios CE2

• Zabbix distribution of item for nodes

• Zabbix worker node checks

• MonaLISA Temperature

Related links

• USLHCNET
• SC

• FTS great new dashboard

Task wishlist

Monitoring requirements

• RSV
  • Cert exists
  • Cert validity

• Nodes
  • GLEXEC
  • CVMFS
  • SWAP Trigger
  • IOPS on HDFS
  • IOPS on /
  • Proc / User
  • Open Files / User

• Servers
  • RAID states

• NameNode/HDFS
  • Health
  • Checkpoints
  • NN - All filesystems usage maximum alert

• External
  • PhEDEx data on transfers
  • Data from SAM
  • Data from RSV
  • Data from DashBoard - Blackhole nodes

• XrootD Monitoring

• XrootD DashBoard

• FTS not-as-good monitoring

• FTS great new dashboard

Monitoring links

Extremely handy links :

MonaLisa HepSpec table

USCMS T2 HepSpec table

Sites pledges

Upgrades (software)

USCMS Upgrades twiki

Pledges, etc

• REBUS
• SiteDB

Monitoring links

We need a page to aggregate those, plus some DashBoard + PhEDEx + central CMS monitoring tools plots.

Central CMS

• Overview
  • Batch system efficiency
  • Site Status Board
  • Site Readiness

• Running production x Pledge
• Pledges and shares
• Grid jobs
  • Daily running
  • Daily pending
  • Weekly running
  • Weekly pending
  • Job efficiency (success/failure)
  • Production jobs efficiency (success/failure)

• Nagios SE
• Nagios CE1
• Transfers, from Caltech
  • Weekly rate
  • Weekly quality

• Transfers, to Caltech
  • Weekly Rate
  • 48h quality

• Perfsonar

• Analysis central monitoring per user

• Subir phedex page

• GlideIn monitoring page

• XrootD DashBoard

OSG

• RSV probes
• Local RSV

Local pages/systems

• Internal batch system monitoring
• Tier-2 hadoop monitoring
• Tier-3 remote hadoop monitoring
• Nagios
• Ticket system
• Campus cacti

Documentation

• Chaging SITECONF settings
• CRAB3 mantra

Workplan 02/2013

T2

• Base improvements in the T2 cluster management schema -- will improve the cluster stability and ease maintenance procedures
• Improvements/integration on the monitoring
  • Automate MonaLisa install into all T2 nodes and servers
  • Clean current MonaLisa web dashboard with the help of the MonaLisa team
  • Clean current Nagios dashboard, reduce to applicable alarm frequency
    • (optional) Implement SMS alerts for Nagios most critical alarms.
• Integrate all new servers into Nagios/Backup schema (could not be done yet)
• Review of network/firewalls/security settings -- more a "learning review"
  • Add few rules so cluster management from CERN is easier
  • (Optional) Explore different SSH authentication methods (GSI or Kerberos) - passwordless but still secure SSH access

T3 (Local & Remote)

• Automate node deployment and profile maintenance with local (dedicated headnode) or remote (T2) puppet+foreman
  • This will make us move away from Rocks in all clusters and have more uniformity in deployment/configuration procedures.
• Commission Condor as a batch system, better monitoring comes for free

Global (related to computing resources in general) :

• https://github.com/dmwm/WMCore/wiki/All-in-one-test[Tutorial]] To install a WMAgent. Let's see.

• Explore how the T3s could use its Idle CPU time (if any) to process jobs for the T2 cluster. (condor integration, condor fork)

• Explore pros/cons of this : The T2 and T3-higgs nodes can store data under the same hadoop namenode, our local resources would be protected by quotas. Management gets simpler

• Explore dynamic Condor job scheduling based on resource utilization and job requirements. Condor will not anymore serve X slots, but will schedule jobs as long as CPU usage is < 90% (configurable) and there is enough available memory in a node. In general this should provide slightly more job slots than the number of cores that a node has. -- motivation : most of the jobs have CPU efficiency < 70%

• Improve BeStman GridFTP selection algorithm (light development needed) -- currently round-robin, very sub-optimal for setup like ours where we have fast (10 GE) servers and slow (1 GE) servers.

CIT HEP Computing

Workplan guidelines

Our priorities are in this order :

• Uptime
• User support (blocker issues)
• Resource utilization -- making sure that all hardware is being used
• User support (non-blocker important issues)
• Backups
• Monitoring
• User support (potential non-issues)
• Security
• R&D - This probably needs extra manpower. We have ideas but no time.

• ComputingT3HiggsFAQ

• ComputingGridTools