Analytics and Business Intelligence systems are important focus area at Inmobi. Hadoop based warehouses operate well over large datasets in the order of TBs and PBs and scales fairly linearly. Queries over Hadoop still typically adopts a full scan approach, which tends to be costly in terms of query turnaround times. On the other end of the spectrum are columnar SQL databases, which lend themselves well for interactive SQL queries over reasonably small datasets in the order of 10-100s of GB. Choosing between these different data stores based on cost of storage, concurrency, scalability and performance are fairly complex and not easy for most users. InMobi had built Lens, (formerly Grill) to solve this very problem.
We at InMobi felt that this is not an exclusive problem for InMobi, but the big data user community at large can benefit from this work. We are happy to announce that Lens has been incubated at Apache Software Foundation..
Apache Lens is an Unified Analytics Platform that cuts data silos. At the core of it, it
- Provides a unified catalog through which schema can be discovered
- Allows users to query a unified datastore without concerning themselves about the physical storage location/structure or the execution engine that actually executes their query
- Manages the query life cycle of user queries and provides multiple ways to access the results
- Provides enterprise ready REST APIs for user authentication, catalog browsing, query lifecycle, query history, query analytics, query completion email-notification.
You can find more details on the project website http://lens.incubator.apache.org/
We invite everyone passionate about big data analytics to contribute and shape Apache Lens into a powerful analytics platform.
Thanks to engineers at InMobi, Hortonworks, Software AG, Flipkart and mentors at The ASF in marking this possible.