Fujitsu Limited · TSE:6702

Fujitsu — Technical Presentation (2011)

18 slides from Fujitsu's technical presentation, 2011

  1. Slide 1: FEFS: Scalable Cluster File System

    FEFS: Scalable Cluster File System K Computer (RIKEN AICS) PRIMEHPC FX10 PRIMERGY "K computer" is the nickname RIKEN has been using for the supercomputer. Copyright 2011 FUJITSU LIMITED

  2. Slide 2: Outline

    Outline  Overview of FEFS  Major features  Target system  I/O Architecture  Concept and system design  High Reliability  Technical Issues  Lustre Extensions  I/O zoning  Fair-share QoS, Best-effort QoS  Performance Evaluation  Throughput (IOR)  Response (mdtest)  Summary  Contribution

  3. Slide 3: Features of FEFS

    Features of FEFS FEFS is a scalable cluster file system based on Lustre. (FEFS: Fujitsu Exabyte File System)  High Performance & High Scalability  Scalable I/O performance (~1TB/s) & capacity (~8EB). Client Nodes Meta Data File Data  I/O Usage Management Meta Data Server  Fair-share QoS (MDS) Ob

  4. Slide 4: Target System

    Target System  K Computer  RIKEN and Fujitsu have been working together to develop the K computer.  To be installed at the RIKEN AICS, Kobe, by 2012  PRIMEHPC FX10  Fujitsu’s brand-new supercomputer recently release.  PC Cluster  PRIMERGY and third-party IA/Linux based servers. Super Computer

  5. Slide 5: System Configuration

    System Configuration Supercomputer PC Cluster • Data transfer to/from global Tofu: 6D mesh/torus Interconnect file system • Data communication for system job operations FEFS: Local file system management (Temporary area occupied by jobs) I/O network (QDR InfiniBand), management network (GbE) User Jo

  6. Slide 6: I/O Architecture: Basic Concept

    I/O Architecture: Basic Concept  Incompatible features is implemented by introducing Layered File System.  Local File System (/work): High Speed FS for dedicated use for jobs.  Global File System (/data): Large Capacity and Redundancy FS for shared use. Thousands of Users Compute Nodes Compute No

  7. Slide 7: I/O Architecture: System Design

    I/O Architecture: System Design  Optimized for Scalable File I/O Operation  Achieving Scalable Storage Volume and Performance  Eliminating I/O Conflicts from Every Components  I/O Zoning Technology for Local File System  File I/O is separated among jobs and processed by I/O node located Z=0. 

  8. Slide 8: Scalable Performance & Capacity

    Scalable Performance & Capacity  High speed throughput and large capacity have achieved by multiple OSSs.  Scale out throughput & capacity by adding servers and storages. OSS OSS Storage Add Server&Storage Throughput&Capacity Number of Server&Storage 7 Copyright 2011 FUJITSU LIMITED

  9. Slide 9: High Reliability and High Availability

    High Reliability and High Availability  Keeping file system service against failures.  Redundant hardware • Duplex paths of InfiniBand, Fibre Channel, I/O Server • RAID disks (MDS：RAID10, OSS: RAID5/6)  System Management software • Detect failure and switch to alternate path or server automatical

  10. Slide 10: Lustre Extension of FEFS: Features

    Lustre Extension of FEFS: Features New FEFS Features Extended Large scale High performance Reuse Max file size File striping MDS response Max number of files Parallel I/O I/O zoning Max client number Max stripe count Client cache 512KB block Server cache OS jitter reduction Network Operations Manage

  11. Slide 11: Lustre Extension of FEFS: Specification

    Lustre Extension of FEFS: Specification Features FEFS Current Lustre System Max file system size 100PB (8EB) 64PB Limits Max file size 1PB (8EB) 320TB Max #files 32G (8E) 4G Max OST size 100TB (1PB) 16TB Max stripe count 20k 160 Max ACL entries 8191 32 Node Max #OSTs 20k 8150 Scalability Max #client

  12. Slide 12: I/O Zoning: I/O Separation among Jobs

    I/O Zoning: I/O Separation among Jobs  Issue: Job’s I/O conflicts on hardware.  Sharing disk volumes, network links among jobs cause I/O performance degradation because of their confliction.  Our Approach: Separate hardware among jobs.  Separating of disk volumes, network links among jobs as muc

  13. Slide 13: QoS: Fair-share QoS

    QoS: Fair-share QoS Issue  Avoiding from some one’s occupying file I/O resources.  Our approach  Limit the number of I/O requests each user can execute simultaneously on the client node. Without Fair Share QoS With Fair Share QoS User A User A Not Fair Fair User B User B Login Node File Servers

  14. Slide 14: QoS: Best-effort QoS

    QoS: Best-effort QoS Issue Utilize all I/O resources effectively. Our Approach Assign all server resources to clients that execute file I/O. Occupied by 1 node Shared by multiple nodes Login node Login node Job Job Job File Servers Compute node Compute node No file I/O 13 Copyright 2011 FUJITSU

  15. Slide 15: Evaluation of FEFS: QoS

    Evaluation of FEFS: QoS QoS efficiency on PC Cluster  User A: 1 node Job ⇒ Measure creation/removal time of 10,000 files.  User B: 19 node Job Influence of User B’s file operation is suppressed. User A’s processing time User A w/o QoS w/o QoS w/ QoS 10,000 Files Single User Multi User Multi User

  16. Slide 16: Summary and Future Works

    Summary and Future Works  Fujitsu developed Lustre based cluster file system FEFS.  High-speed file I/O (~1TB/s), Huge capacity (~8EB)  High-reliability and High-availability  Luster enhancements: QoS, IB multi-rail, directory Quota.  Future Works  Contribute our efforts to the Lustre communit

  17. Slide 17: Press Release

    Press Release 16 Copyright 2011 FUJITSU LIMITED

  18. Slide 18: Copyright 2010 2011 FUJITSU LIMITED

    17 Copyright 2010 2011 FUJITSU LIMITED

Fujitsu — Technical Presentation (2011) on 6ix