Experimental evaluation of a flexible I/O architecture for accelerating workflow engines in ultrascale environments Articles uri icon

publication date

  • January 2017

start page

  • 52

end page

  • 67

volume

  • 61

International Standard Serial Number (ISSN)

  • 0167-8191

Electronic International Standard Serial Number (EISSN)

  • 1872-7336

abstract

  • The increasing volume of scientific data and the limited scalability and performance of storage systems are currently presenting a significant limitation for the productivity of the scientific workflows running on both high-performance computing (HPC) and cloud platforms. Clearly needed is better integration of storage systems and workflow engines to address this problem. This paper presents and evaluates a novel solution that leverages code sign principles for integrating Hercules an in-memory data store with a workflow management system. We consider four main aspects: workflow representation, task scheduling, task placement, and task termination. The experimental evaluation on both cloud and HPC systems demonstrates significant performance and scalability improvements over existing state-of-the-art approaches. (C) 2016 Elsevier B.V. All rights reserved.

keywords

  • workflow; i/o acceleration; high-performance computing; cloud computing; big data; system