Trifork Group AG · OMXCOP:TRIFOR

Trifork — Zanox — Technical Presentation (September 2013)

12 slides from Trifork's technical presentation, Sep 6, 2013

  1. Slide 1: Denormalizing Data Sets in Hadoop While Building Lucene Indexes

    DENORMALIZING DATA SETS IN HADOOP WHILE BUILDING LUCENE INDEXES Zürich | 06.09.13

  2. Slide 2: Table of Contents

    TABLE OF CONTENTS 1. ABOUT ZANOX AND ME 2. REPORTING IN GENERAL 3. MAP & REDUCE SIDE JOINS 4. QUESTIONS Zürich | 06.09.13 | zanox | Zanox Reporting Systems 2

  3. Slide 3: The Zanox Network

    THE ZANOX NETWORK Zürich | 06.09.13 | zanox | Zanox Reporting Systems 3

  4. Slide 4: Who Am I?

    WHO AM I? ● Senior Architect at Zanox ● 7+ years experience in building distributed search systems based on Lucene Lucid Certified Apache Solr/Lucene Developer ● 5+ years of using Hadoop and 2+ years of using HBase for data mining Cloudera Certified Developer for Apache Hadoop and HBase ● I have app

  5. Slide 5: About Zanox and Me

    1. ABOUT ZANOX AND ME 2. REPORTING IN GENERAL 3. MAP & REDUCE SIDE JOINS 4. QUESTIONS Zürich | 06.09.13 | zanox | Zanox Reporting Systems 5

  6. Slide 6: Map & Reduce Side Joins

    REPORTING IN GENERAL 00:00 03:00 06:00 09:00 12:00 15:00 18:00 21:00 01:00 02:00 04:00 05:00 00:00:00 00:02:00 00:04:00 00:06:00 00:08:00 00:10:00 00:10:20 00:10:40 00:11:00 00:11:20 00:11:40 00:12:00 00:14:00 00:16:00 00:18:00 00:20:00 00:22:00 00:24:00 00:26:00 00:28:00 00:00:00.000 00:00:00.005 0

  7. Slide 7: Reporting in General

    1. ABOUT ZANOX AND ME 2. REPORTING IN GENERAL 3. MAP & REDUCE SIDE JOINS 4. QUESTIONS Zürich | 06.09.13 | zanox | Zanox Reporting Systems 7

  8. Slide 8: Map & Reduce Side Joins: Map-Side

    MAP & REDUCE SIDE JOINS Map-Side Reduce-Side Both-Side Advantages Reducers are not needed for joining MR pipeline can be used for other purposes INITIALISE MAP TASKS BY Extremely Mapfriendly regarding network DISTRIBUTING SMALLER DATASET Distributed cache supports distribution of dataset that should

  9. Slide 9: Map & Reduce Side Joins: Reduce-Side

    MAP & REDUCE SIDE JOINS Map-Side Reduce-Side Both-Side Advantages Datasets of any size can be joined Map Easy to implement because MP helps Reduce Disadvantages Loads significantly network because both datasets need Mapto be completely transferred to reducers Keys that mappers generate need to be ta

  10. Slide 10: Map & Reduce Side Joins: Both-Side

    MAP & REDUCE SIDE JOINS Map-Side Reduce-Side Both-Side Advantages Frequent join keys are handled on map-side Reducers are needed only for not frequent join keys Map Consequently it is extremely network friendly Skew in join keys is fixed because infrequent keys Reduce cannot make significant imbalan

  11. Slide 11: Map & Reduce Side Joins

    1. ABOUT ZANOX AND ME 2. REPORTING IN GENERAL 3. MAP & REDUCE SIDE JOINS 4. QUESTIONS Zürich | 06.09.13 | zanox | Zanox Reporting Systems 11

  12. Slide 12: Questions

    QUESTIONS Zürich | 06.09.13 | zanox | Zanox Reporting Systems 12

Zanox — Technical Presentation (September 2013) on 6ix