Tonglin Li | Illinois Institute of Technology (original) (raw)

Papers by Tonglin Li

We have built a new framework that enables independent metadata updates within HDF5 files. Our fr... more We have built a new framework that enables independent metadata updates within HDF5 files. Our framework includes the following components: an overlay network that's optimized for 1-to-All and All-to-All communication; root-less/server-less Bcast and AllReduce operations for reduced synchronization; a distributed consensus-making platform; an execution order keeping system, and a progress engine for generic threadless concurrence handling.

Bookmarks Related papers MentionsView impact

H5bench benchmark suite contains a list of applications that are used to measure the HDF5 I/O per... more H5bench benchmark suite contains a list of applications that are used to measure the HDF5 I/O performance from various aspects.

Bookmarks Related papers MentionsView impact

Abstract—With the increased scale of systems in use and the need to quickly store and retrieve in... more Abstract—With the increased scale of systems in use and the need to quickly store and retrieve information, key/value stores are becoming an important element in the design of large-scale storage systems. Key/value stores are well known for their simplistic interfaces, persistent nature, and excellent operational efficiency – they are also known as NoSQL databases. This paper presents the design and implementation of a non-volatile hash table (NoVoHT). NoVoHT was designed from the ground up to be lightweight, fast, and dependency-free. Our goal was to create a fast persistent key/value store that could be easily integrated and operated in lightweight Linux OS typically found on today’s supercomputers. We also aimed to develop a system that performed as close as possible to an in-memory hash map, but with the added benefit of being persistent. We also extended the traditional key/value store interface (e.g. insert, lookup, remove) to include a novel operation (e.g. append) that has a...

Bookmarks Related papers MentionsView impact

computing

Bookmarks Related papers MentionsView impact

ZHT is a zero-hop distributed hash table, which has been tuned for the requirements of high-end c... more ZHT is a zero-hop distributed hash table, which has been tuned for the requirements of high-end computing systems. ZHT aims to be a building block for future distributed systems. The goals of ZHT are delivering high availability, good fault tolerance, high throughput, and low latencies, at extreme scales of millions of nodes. ZHT has some important properties, such as being lightweight, dynamically allowing nodes to join and leave, fault tolerant through replications, persistent, scalable, and supporting unconventional operations such as append. ZHT scaled up to 32K-cores with latencies of 1.1ms and 18M operations/sec throughput on IBM Blue Gene/P supercomputer, and 96 nodes on Amazon EC2 cloud with 800ns latency and 1.2M ops/s throughput. In previous work we proved ZHT’s excellent performance and scalability on supercomputers, and in this work we show that it also works great on cloud environment from both performance and cost perspective.

Bookmarks Related papers MentionsView impact

Over the last decade, storage systems have experienced a 10fold increase between their capacity a... more Over the last decade, storage systems have experienced a 10fold increase between their capacity and bandwidth. This gap is predicted to grow faster with exponentially growing concurrency levels, with future exascales delivering millions of nodes and billions of threads of execution. A critical component of future file systems for high-end computing is metadata management. This extended abstract presents ZHT, a zero-hop distributed hash-table, which has been tuned for the specific requirements of high-end computing. The primary goal of ZHT is excellent availability, fault tolerance, high throughput, and low latencies. 1.

Bookmarks Related papers MentionsView impact

Today with the rapidly accumulated data, datadriven applications are emerging in science and comm... more Today with the rapidly accumulated data, datadriven applications are emerging in science and commercial areas. On both HPC systems and clouds the continuously widening performance gap between storage and computing resource prevents us from building scalable data-intensive systems. Distributed NoSQL storage systems are known for their ease of use and attractive performance and are increasingly used as building blocks of large scale applications on cloud or data centers. However there are not many works on bridging the performance gap on supercomputers with NoSQL data stores. This work presents a convergence of distributed NoSQL storage systems in clouds and supercomputers. It firstly presents ZHT, a dynamic scalable zero-hop distributed key-value store, that aims to be a building block of large scale systems on clouds and supercomputers. This work also presents several real systems that have adopted ZHT as well as other NoSQL systems, namely ZHT/Q (a Flexible QoS Fortified Distribute...

Bookmarks Related papers MentionsView impact

Today’s science is generating datasets that are increasing exponentially in both complexity and v... more Today’s science is generating datasets that are increasing exponentially in both complexity and volume, making their analysis, archival, and sharing one of the grand challenges of the 21st century. Exascale computing, i.e. 10 FLOPS, is predicted to emerge by 2019 with current trends. Millions of nodes and billions of threads of execution, producing similarly large concurrent data accesses, are expected with the exascale. Current state-of-the-art yet decades long storage architecture of high-performance computing (HPC) systems would unlikely provide the support for the expected level of concurrent data access. The main critique comes from the topological allocation of compute and storage resources that are interconnected as two cliques. Even though the network between compute and storage has high bandwidth and is sufficient for compute intensive petascale applications, it would not be adequate for data-intensive petascale computing or the emerging exascale computing (regardless if it...

Bookmarks Related papers MentionsView impact

Although the traditional relational database has been used and dominated for many year, the limit... more Although the traditional relational database has been used and dominated for many year, the limitation of it has appeared with the huge number of connected data which is generated by today’s Internet, Web2.0 and social networks. The query operation will be tremendously slow to query by the traditional relational database. The state-of-the-art graph database is usually Master/Slave architecture thus these systems cannot achieve high scalability and they will become harder and harder to handle the huge number of connected data. In this paper we have design and implement a graph database using ZHT as a block. This graph database is also following the principle of Bulk Synchronous Parallel(BSP) model. We have overcome several problems of using ZHT as basic component in BSP model. We will give the information of our design and implementation in the following part. At last we will also give a comprehensive performance evaluation. Keywords—ZHT; Graph Database; Graph Processing System;

Bookmarks Related papers MentionsView impact

The interoperability across multiple or many blockchains would play a critical role in the forthc... more The interoperability across multiple or many blockchains would play a critical role in the forthcoming blockchain-based data management paradigm. In particular, how to ensure the ACID properties of those transactions across an arbitrary number of blockchains remains an open problem in both academic and industry: Existing solutions either work for only two blockchains or requires a centralized component, neither of which would meet the scalability requirement in practice. This short paper shares our vision and some early results toward scalable cross-blockchain transactions. Specifically, we design two distributed commit protocols and, both analytically and experimentally, demonstrate their effectiveness.

Bookmarks Related papers MentionsView impact

It has become increasingly important to capture and understand the origins and derivation of data... more It has become increasingly important to capture and understand the origins and derivation of data (its provenance). A key issue in evaluating the feasibility of data provenance is its performance, overheads, and scalability. In this paper, we explore the feasibility of a management layer for parallel file systems, in which metadata includes both file operations and provenance metadata. We design and implement a provenance layer within a distributed file system —FusionFS, which implements a distributed file metadata management based on distributed hash tables. Our results show that FusionFS with its own storage layer for provenance capture is able to scale up to 1K nodes on BlueGene/P supercomputer.

Bookmarks Related papers MentionsView impact

Computers & Electrical Engineering

Bookmarks Related papers MentionsView impact

IEEE Transactions on Big Data, 2017

In Big Data era, applications are generating orders of magnitude more data in both volume and qua... more In Big Data era, applications are generating orders of magnitude more data in both volume and quantity. While many systems emerge to address such data explosion, the fact that these data’s descriptors, i.e., metadata, are also “big” is often overlooked. The conventional approach to address the big metadata issue is to disperse metadata into multiple machines. However, it is extremely difficult to preserve both load-balance and data-locality in this approach. To this end, in this work we propose hierarchical indirection layers for indexing the underlying distributed metadata. By doing this, data locality is achieved efficiently by the indirection while load-balance is preserved. Three key challenges exist in this approach, however: first, how to achieve high resilience; second, how to ensure flexible granularity; third, how to restrain performance overhead. To address above challenges, we design Dindex, a distributed indexing service for metadata. Dindex incorporates a hierarchy of coarse-grained aggregation and horizontal key-coalition. Theoretical analysis shows that the overhead of building Dindex is compensated by only two or three queries. Dindex has been implemented by a lightweight distributed key-value store and integrated to a fully-fledged distributed filesystem. Experiments demonstrated that Dindex accelerated metadata queries by up to 60 percent with a negligible overhead.

Bookmarks Related papers MentionsView impact