Appistry solutions leverage cutting-edge cloud-based architectures
Cloud-based architectures, with their inherent distribution of storage and computation, provide an ideal foundation for large-scale analytics, where many gigabytes or terabytes of data must be stored and processed. By designing solutions that leverage the inherent scalability, capacity, performance, simplicity and cost-efficiency of cutting-edge cloud technology, we help companies access computational power unlike any currently available.
Our cloud-based architectures unify large quantities of affordable, commodity systems with directly-attached storage, all working in concert to provide you with a single system view that transcends the performance and capacity of any single machine within the system.
The result is a system that combines three core technologies into a single unified platform. First, our system is a High Performance Computing system. Second, it is a Cloud Computing platform. And third, it is a complex analytics platform. Appistry combine all three together with unified storage and computation in a system of unlimited flexibility and power at a truly affordable price.
Appistry’s cloud-based analytics solutions give you:
- Performance — Distributed processing and data agility can easily deliver 10-100x performance gains over “big iron” deployments
- Scalability — An administrator is able to add additional computers, including their storage capacity, to a running system without a loss of availability of files or administrative functionality. Because tracking and membership are fully distributed and dynamic, the overall system can grow to tens of thousands of systems, or more.
- Capacity — The analytics system provides a global view or namespace, aggregating the compute and storage capacity of all attached servers.
- Reliability — By allowing the user to specify how many copies of each file to maintain in the system, the system is able to offer high levels of reliability at low cost.
- Geographic Distribution — A single instance of a cloud-based analytics system can be deployed across multiple data centers. The cloud is aware of the network topology and will mirror and distribute files across the network so that the loss of any one data center does not limit access to data.
- Disaster Recovery — The system is fully distributed; there is no central point of failure. Data ingest and analysis can continue operation even when entire data centers have been removed from the system.
- Availability — Every computer in the analytics system is capable of performing analytics computations, managing data and responding to administrative requests. As a result, the system is impervious to the loss of individual or even entire racks of machines.
- Management Simplicity — Administrators are able to update computer configurations, system configurations, or any of the running analytics applications without taking the files offline.
- Hardware Flexibility — Not all machines in the system need to be constructed from similar hardware. The system can recognize the attributes of each attached computer and utilize their resources accordingly.
High-resolution satellites, multimodal sensors and other input sources are driving an explosion in data available to the Intelligence community. This presents a data processing challenge.
Ayrris™ / DEFENSE overcomes these challenges by providing high-volume data ingest, storage, analysis and delivery. By leveraging Appistry’s revolutionary Computational Storage™ technology, Ayrris
turns a cluster of commodity servers into a high-performance, petabyte-scale distributed storage system with no single points of failure or I/O bottlenecks. Rather than move the data to the application, we prefer to move the application to the data. Because of its unique computing platform, Ayrris / DEFENSE offers a new level of scalability, elasticity and reliability for dataintensive applications, and is fully compatible with existing agency data sources and analysis tools. Ayrris / DEFENSE allows enterprises to quickly turn raw data into useable, mission-critical intelligence better, faster and cheaper than ever before.
The following three technology trends are having a dramatic impact on the way big data challenges will be addressed:
- Transitioning Storage Systems to Cloud Technologies
- Commoditization of Storage Equipment
- Move Towards Data Localization
Industry progress in these areas provides solutions for the construction of large data storage systems.
Cloud computing architectures, on the other hand, are characterized by their use of large quantities of affordable, commodity systems with directly-attached storage, all working in concert to provide the user with a single system view that transcends the performance and capacity of any single machine within the system. A storage system
built in this manner provides the following attributes:
Scalability. A cloud storage administrator is able to add additional computers, including their storage capacity, to a running system without a loss of availability of files or administrative functionality. Because tracking and membership are fully distributed and dynamic, the overall system can grow to tens of thousands of systems, or more.
Capacity.The cloud storage system provides a global view or namespace, aggregating the capacity of all attached storage devices.
Reliability. Cloud storage allows the user to specify how many copies of each file to maintain in the system. The cloud is aware of the loss of any machines in the system. When these errors occur, the cloud can alert the proper administrators and take appropriate action to recover the requested reliability level.
Geographic Distribution. A single instance of a cloud storage system can be deployed across multiple data centers. The cloud is aware of the network topology and will mirror and distribute files across the network so that the loss of any one data center does not limit access to data.
Disaster Recovery. The storage system is fully distributed; there is no central point of failure. Cloud storage can continue operation even when entire data centers have been removed from the system. Cloud storage also manages the merging of multiple data centers after a logical or physical separation occurs. Out-of-date files are located and reconciled without user-intervention whenever possible.
Availability. Every computer in the cloud system is capable of serving access to files or administrative requests. Cloud storage is easily able to service a large number of client requests by distributing the work across many machines. The system is impervious to the loss of individual or even entire racks of machines.
Manageable. Administrators are able to update computer configurations, system configurations, or update the cloud system itself without taking the files offline.
Heterogeneous. Not all machines in the cloud system need to be constructed from similar hardware. The system needs to recognize the attributes of each attached computer and utilize their resources accordingly.
By taking a cloud-oriented approach to storage and compute, we are able to deliver a more powerful system. Moreover, because cloud storage systems are built with commodity components, they are much less expensive than traditional approaches.
Historically, system architects and administrators have depended on increasingly larger and larger machines and devices to satisfy their growing computational and storage needs. These high-end, proprietary systems have come at a steep price in terms of both capital and operational costs, as well as in terms of agility and vendor lock-in. The advent of storage and computational systems based on cloud architectures results in an advantageous economic position for purchasers and users of these solutions.
Move Towards Data Localization
In traditional system architectures, computational elements (i.e. application servers) and storage devices are partitioned into separate islands, or tiers. Applications pull data from storage devices via the local or storage area network, operate on it, and then push the results back to the storage devices. As a result, for most traditionally architected applications, the weak link in the system is the bottleneck between the application and its data.
Data localization is the unification of storage devices with computational elements for the purposes of reducing computational latency and overcoming network bottlenecks.
In a system exhibiting data locality, the work is moved to the data instead of the data being moved to the work. CloudIQ Storage was built from the ground up to enable data localization, which Appistry calls computational storage™. Other examples of data localization in practice include the Apache Hadoop project—an implementation of the MapReduce algorithm initially popularized by Google, Netezza’s data warehouse appliances, and various data caching technologies.
One way to compare the relative performance of traditional and cloud storage approaches and to quantify the performance benefits of computational storage is to look at the aggregate bandwidth available between application processing and storage.
Taken together, the impact of cloud computing architectures, the commoditization of storage and compute, and the move towards data localization are revolutionizing the delivery of data-intensive applications; solving problems once thought to be unsolvable because of economic or temporal concerns has now become possible.
Appistry CloudIQ Storage: Cloud Technology Applied to Big Data
Appistry CloudIQ Storage applies cloud computing architectural principles to create a scalable, reliable and highly cost-effective file storage system with no single points of failure, using only commodity servers and networking.
A CloudIQ Storage system is composed of multiple computers at one or more data centers,
CloudIQ Storage coordinates the activity of each of the computers and aggregates their attached storage to expose a single, logical data store to the user. The system is fully decentralized: each computer is a complete CloudIQ Storage system unto itself, but is aware of other members of the same storage system and shares storage responsibilities accordingly.
Several of the major functional characteristics of the CloudIQ Storage system are below. We consider these to be essential architectural characteristics of any cloud-based storage system.
Self-Organizing and Scalable
Appistry believes that fully distributed systems with self-healing and self-organizing properties are the path to solving big data challenges. The foundation of the CloudIQ storage architecture is a lightweight, yet robust membership protocol. This protocol updates the member machines with the addition, removal or unexpected loss of computers
dedicated to the storage system. This shared membership data contains enough information about the members of the cloud that each individual machine is capable of assessing its location and responsibilities. These responsibilities include establishing
proper communication connections and responding to system events requiring healing actions. Even though there is no central control structure, the system is capable of selforganizing thousands of machines.
An administrator can easily add or remove machines or update configurations. The members of the cloud, acting independently, will share information quickly and reconfigure appropriately. The storage cloud can elastically scale up to handle multiple petabytes without heavy management overhead.
One desired feature of a robust storage system is location awareness. Computers within the CloudIQ Storage environment can use location awareness to make informed decisions about reliability configurations and to optimize the handling of user requests.
CloudIQ Storage introduces the notion of a territory to be a logical collection of machines classified as a group for the purpose of data distribution. Users typically assign territories in one of several ways:
Computer Rack or Network Switch. This configuration allows an administrator to instruct a storage cloud to distribute files and functionality across systems within a single data center.
Data Center. This configuration allows an administrator to inform the cloud to distribute files and functionality between data centers.
User-Based. For storage clouds that span multiple geographies, it is beneficial to inform the system which computers are dedicated to individual user groups. Often this is a similar configuration to the data center option.
Hardware-Based. This configuration allows different configurations of hardware to be grouped together. These groups provide the administrator with a method to store data on specialized hardware for different needs. For example, within a data center one might have two territories of low-latency computers set up across racks for availability. A third collection of machines might be constructed of higherstorage- density, higher-latency hardware to keep costs low while maintaining a third copy of the data.
Territory settings can be configured on a machine-by-machine basis. Administrators can choose from any of these use cases or develop hybrid configurations that meet their needs.
CloudIQ Storage uses territory settings to implement the behaviors described in the remainder of this section.
CloudIQ Storage provides high levels of reliability by distributing files and their associated metadata throughout the storage system. Each copy of a file possesses audit and configuration information needed to guarantee reliability requirements are achieved. The metadata of each file contains information on:
Reliability Needs. How many copies of a file need to be maintained?
Territory Distribution. Which territories can/should be used to keep a copy of the files?
Update History. What is the version history of each file?
The reliability requirements of each file in the system are distributed across the machines in the CloudIQ Storage system. Each machine watches over a subset of files in the system. When these monitors detect system changes, the following actions occur to guarantee the reliability needs of the system:
File Reliability Checks. Each monitor examines the files for which it is
responsible. If a machine holding a copy of the file has been lost, additional copies of the file are created.
File Integrity Checks. If a dormant or lost machine attempts to introduce an old copy of an existing file, the system reconciles the version against the metadata of the newer files and acts to reconcile the difference.
System Monitoring Reconfiguration. As machines are introduced or lost, the responsibilities for watching files are adjusted for the new system configuration.
File Placement Reconfiguration. As new machines become available, the monitors may decide to redistribute files. The reconfiguration distributes the storage needs and service requests more equally across machines in the storage cloud.
Files may also need to be repositioned to meet territory placement requirements.
As the storage cloud grows and changes with new hardware, new network connections, and configuration changes, the cloud storage system will constantly maintain the proper file distribution and placement.
CloudIQ Storage provides extraordinary levels of availability due to the completely decentralized nature of the architecture. Every computer within the system is capable of serving file ingestion or file retrieval requests. Therefore, the total bandwidth in and out of the system is the aggregate of that of all of the machines participating in the cloud.
In addition, even though multiple copies of the file are present in the cloud storage system, the user gets only a single, logical view of the system. The CloudIQ Storage architecture resolves the multiple territories, copies and even versions of each file to deliver the requested file to the user.
When a file retrieval request arrives at a computer in a cloud storage system, several actions occur to provide the user with their data quickly:
File Location. The computer servicing a file request locates which machines in the cloud hold the requested file using consistent hashing and a distributed hash tables.
No single machine holds the entire file directory, as it would become a performance bottleneck or a point of failure. Lookups are a constant time operation that returns the machines within the system holding a copy of the file.
Machine Selection. Once the target machines holding the file have been identified, the requesting machine can choose which machine is optimal for retrieving the file.
This choice can be made based on factors such as network proximity and machine utilization.
File Retrieval. Once the machine is selected, the file can be retrieved by the client.
In addition to optimized read operations, the cloud storage solution provides “always writable” semantics using a concept called eventual consistency. In an eventually consistent system, write operations always succeed as long as the system can access the number of nodes required by policy for a successful write (one, by default). During this write operation, audit information is stored with the file’s metadata so that any additional copies or version reconciliation can be performed later. Eventually consistent systems are not right for every storage application, but it is ideal for “write once, read many” style systems.
The availability, reliability, and location awareness features of a cloud storage solution bring the highest level of disaster recovery available to a storage administrator.
The system can lose a machine, a rack, or even an entire data center and the system remains capable of performing all necessary operations for the user.
Management and ease-of-use features are essential for the creation of a robust cloud storage system. When dealing with hundreds or thousands of machines, management operations must be simplified. CloudIQ Storage ensures this by providing the following
Always Available Operation. Any configuration changes made to the system must not remove the availability of files. In the event that multiple machines need to be taken off line for updates, the system must have a strategy for keeping files available. This may be achieved using territories. If two territories hold copies of the same files, machines in one territory can temporarily be taken off-line for updates
and the second territory can serve the files. Any file updates performed during the downtime will be automatically reconciled using the monitoring and reliability features of the cloud.
Configurable Reliability Settings. Administrators can declare how many copies of a file should be stored in the storage cloud. A cloud-wide setting is established, which may be overridden on a file-by-file basis.
Real-Time Computer Injection. When the storage system needs more capacity, the administrator needs to be able to add machines without affecting the availability of any file.
Real-Time Computer Decommissioning. When it is decided that a computer is no longer required, the administrator needs operations to gracefully remove the computer from processing requests and move its files to other machines within the cloud.
Auditing. Important operations, events, and system messages need to be saved.
System-Wide Configuration Changes. All configuration changes need to propagate across all machines with a single operation.
Because management tasks in the storage cloud are virtualized and automated, a small number of system administrators can easily maintain a large number of computers storing petabytes of data.
CloudIQ Storage implements a flexible security model designed to allow multi tenant operation while ensuring the security of user data. Just as the physical storage cloud
may be partitioned into territories, the logical storage cloud may be partitioned into “spaces,” with each space representing a hierarchical collection of files under common control. Access to each space, and to each file within a space, is governed by an access control list (ACL) that assigns access rights for users, groups and the file’s
To facilitate secure operation of the cloud, administration rights to the system are divided into a series of distinct privileges that may be assigned to individual users or groups.
Traditional storage and computational offerings fail to meet the needs of today’s big data environments. These approaches have been characterized by isolated pools of expensive storage and the constant movement of data from where it lives to the application servers and users who need it. Attempting to meet demanding capacity and
performance requirements in this manner often necessitates the purchase of extremely costly, special-purpose hardware. Yet, local and wide-area network bottlenecks remain a challenge.
Petabyte-scale environments dictate the need for distributed, high-performing solutions that bring the work to the data, not the other way around. In this paper we have demonstrated how the cloud computing architectural approach provides the key to meeting the challenges posed by big data.
Appistry CloudIQ Storage is software that applies these principles to deliver robust private cloud storage environments. Storage clouds based on CloudIQ Storage exhibit essential characteristics Appistry propose for any cloud storage system: individual resources self-organize without centralized control, yielding extreme scalability; the system spans data centers and optimizes behavior based on topology; high levels of reliability and availability are transparently ensured; system management is policy-based, automated and virtualized.
With the Hadoop Edition, Appistry hopes to “upgrade” the performance and availability of Hadoop-based applications by replacing the Hadoop Distributed File System (HDFS) with CloudIQ Storage. While Hadoop is wildly popular right now, one issue is its use of a “namenode” – a centralized metadata repository that can constrain performance and creates a single point of failure. Appistry’s approach retains Hadoop’s MapReduce engine to assign parallel-processing tasks, but attempts to resolve namenode problems with CloudIQ Storage’s wholly distributed architecture.
Appistry is with an intelligence-sector customer that has “massive, massive” applications built on HBase, a distributed NoSQL database with Hadoop at its core. Although CloudIQ Storage doesn’t formally support HBase, it has helped the customer improve database throughput, and formal support might be on the way. Because of their inherently scalable natures, CloudIQ Storage and NoSQL databases are complementary solutions to handle structured and unstructured data.
The idea behind cloud storage is the same as the idea behind cloud computing: Organizations want to meet their ever-expanding storage needs as they arise, and they want to do so at lower price points than are available from incumbent vendors like EMC and NetApp. For customers in areas like social media, scientific imaging or film rendering, though, scale and price must be matched with performance. This is where companies like Appistry come in, but it certainly isn’t alone in the quest for these dollars. Startups Scale Computing, Pivot3, MaxiScale and ParaScale all have raised millions for their unique offerings, and HP last summer snatched IBRIX to boost its relevance in the performance-hungry film-rendering market.