Aspera on demand for AWS (S3 inc) overview

Aspera OnDemand for AWS
S3|Direct
Bhavik Vyas - Director of Cloud Services & Partnerships
V3.feb2012

Aspera Corporate Overview

• Headquarters: Emeryville, CA
• Founded: 2004
• Ownership: Privately Held
• Employees: 95
• Patents: Issued and pending in over 32 countries
• Customers: over 1,200 customers, over 10,000 licenses
• Markets Served: Media and Entertainment, Federal Government, Life Sciences, Healthcare, Cloud
Computing, Software and Gaming, Financial Services, Legal, eDiscovery, Engineering, Technology,
Telecommunications, Service Providers, Architecture and Design, Enterprise IT

• Creators of the fasp™ protocol
– Innovative, patented, highly efficient bulk data transport technology
– Unique and core to Aspera’s high-performance file transfer software suite
– Outperforms software and hardware WAN acceleration solutions
– Ranked first in every WAN transfer throughput benchmark

Creating next-generation transport technologies that move the world’s digital assets at maximum speed,
regardless of file size, transfer distance and network conditions.

What Constitutes “Big Data” at Aspera?

How Big is “Big Data”? (M&E example)

A single digital cinema production can be 800K–1M 2K/4K frames

Data Transport Challenges with TCP and alternative technologies

• Distance degrades conditions on all networks
– Latency (or Round Trip Times) increase
– Packet losses increase
– Fast networks just as prone to degradation

• TCP performance degrades with distance
– Throughput bottleneck becomes more severe with
increased latency and packet loss

• TCP does not scale with bandwidth
– TCP designed for low bandwidth
– Adding more bandwidth does not improve throughput

• Alternative Technologies
– TCP-based - Network latency and packet loss must be low TCP powers:
– Modified TCP - Improves on TCP performance but insufficient for fast networks • FTP
– UDP traffic blasters - Inefficient and waste bandwidth • SCP
– Data caching - Inappropriate for many large file transfer workflows • RSYNC
• HTTP(s)
– Data compression - Time consuming and impractical for certain file types
• CIFS
– CDNs & co-lo build outs - High overhead and expensive to scale • NFS

What Happened to my Bandwidth?

WAN

1000 Mbps
– 170ms RTT
Seattle – 0.001% packet loss rate Paris

WAN Throughput is 1000Mbps
Max TCP Throughput ~29Mbps
Where’s my 971Mbps?
At 29Mbps:
50GB transfers will take 4 hrs
1TB transfers will take 3.3 days

Aspera’s Solution: high-performance data transport (fasp™)

• Maximum line-rate WAN transfer speed
– Optimal end-to-end throughput efficiency
– Transfer performance scales with bandwidth independent of
transfer distance and resilient to packet loss
• Congestion Avoidance and Policy Control
– Automatic, full utilization of available bandwidth
– On-the-fly prioritization and bandwidth allocation
• Uncompromising security and reliability
– Secure, user/endpoint authentication
– AES-128 cryptography in transit & at-rest
• Scalable management, monitoring and control
– Real-time progress, performance and bandwidth utilization
– Detailed transfer history, logging, and manifest
• Enterprise-Class File Delivery
– Transfers up to thousands of times faster than FTP
– Precise and predictable transfer times
– Extreme scalability (concurrency and throughput)

Same WAN Scenario with Aspera

WAN

1000 Mbps
– 170ms RTT
– 0.001% packet loss rate
Seattle Paris
TCP
Aspera
fasp

WAN is 1000Mbps
• Max TCP Throughput ~29Mbps
• Max Aspera Throughput ~995Mbps (gain of x34)
• ROI measured in $$ cost of not using 710Mbps

At 995 Mbps
– 50GB transfer will take ~4 hrs – 1TB transfer will take 3.3 days
– 50GB transfer will take ~7 mins – 1TB transfer will take 2.4 hrs

fasp TM – performance breakthrough

FTP Across US US – EU US – ASIA Satellite
1 GB 1 – 2 hrs 2 – 4 hrs 4 – 20 hrs 8 – 20 hrs
10 GB 15 – 20 hrs 20 – 40 hrs Impractical Impractical
100 GB Impractical Impractical Impractical Impractical

TCP transfer times limited by packet loss, delay (network distance) NOT BW

fasp™ 2 Mbps 10 Mbps 45 Mbps 100 Mbps 200 Mbps 1 Gbps

1 GB 70 min. 14 min. 3.2 min. 1.4 min. 42 sec. 8.4 sec.
10 GB 11.7 hrs 140 min. 32 min. 14 min. 7 min. 1.4 min.
100 GB 23.3 hrs 5.3 hrs 2.3 hrs 1.2 hrs 14 min.

Aspera transfer times shorten linearly with bandwidth
Independent of packet loss, delay (network distance)
Cross US – Add 1to 5% Intercontinental – Add 1 to 10% Satellite – Add 1 to 10%

Aspera software product & technology portfolio

Distribute Collaborate Automate

Complete portfolio of servers and end point Global person-to-person and project-based Web-based application and SDK for creating and
clients for high-speed digital content delivery and exchange and collaboration of files and directories, managing automated workflows, from simple file
distribution. of any size, over any distance, over any network. forwarding, to complex process orchestration.

Enterprise and Connect Server faspex Server Orchestrator
• Universal file transfer server and web-based • Secure digital delivery and collaborative file • Intuitive graphical workflow designer
interface and directory listing transfers with remote users and partners • File processing decision tree and flow
Client and Point-to-point • Integrated e-mail notifications for delivery and • Rich and flexible plug-in architecture for third-
• Uni- and bi-directional transfer clients successful download party process integration
• Comprehensive administration, user • Comprehensive library of plug-ins for
Connect
management & access control transcoding, virus checking, quality checking,
• Web browser plug-in for high-speed uploads
faspex Multi-Server / HA archive, notifications
and downloads
• Automated bi-directional relays between sites • High volume processing
Mobile • Detailed dashboard, workflow, and step-level
and multiple servers
• High-speed transfer for mobile devices progress reporting.
• 3-tier architecture with support for clustering and
Sync high availability • Open development framework for designing
• Highly scalable, multidirectional file replication and integrating highly processing and
Cargo
and synchronization automation pipelines
• Automated client downloads

Transport
Our unique, patented transport technologies provide unparalleled speed, efficiency, concurrency and bandwidth control over any size, distance, and network

fasp™ fasp3™ Aspera On-Demand S3|Direct
Patented, file-based bulk data transport Next-gen protocol for any bulk data High-speed transfer direct to cloud storage (S3)
fasp-AIR™ fasp-MC™ Console transport management
Uploads and downloads over 3G, LTE and Wi-Fi networks High-speed delivery over multicast Centralized web-based management, monitoring, and reporting

fasp: Patented Bulk-Data Transport Options

faspTM
• Aspera’s original, unique, patented transport for moving bulk data at maximum speed
• Independent of file size, distance, or network type
• Independent of network delay and robust to extreme packet loss
• Built in adaptive rate control achieves extraordinary quality of service and predictable delivery times
• Complete security and data integrity verification

fasp3TM
• Aspera’s 3rd generation patented transport for moving any bulk data, not just files

fasp-AIRTM
• Designed and optimized for high-speed uploads and downloads over 3G, LTE and Wi-Fi networks.
• Supports Android and iOS devices
• SDK allows app developers to integrate superior transport capabilities into their own applications

fasp-MCTM
• Enables high-speed transport over IP Multicast
• Shortest end-to-end distribution time
• Eliminates any pre- and post-transmission delays
• Unrivaled repair efficiency and network utilization

Aspera Console: File Management & Reporting

• Centralized network-wide management and transfer monitoring
• Track and manage all Aspera transfers
throughout the enterprise
• Remote configuration all managed Aspera Server nodes
• Create and manage users & user groups
• Initiate and manage transfer
• Precise control of transfer & bandwidth utilization parameters
• Comprehensive dashboard reporting:
• Real-time transfers and bandwidth utilization
• Transfer event notifications
• Full transfer logging
• Extensive custom reporting capabilities

Aspera Distribute

Complete portfolio of servers and end point clients for high-speed digital content delivery and distribution.

Client and Point-to-Point Enterprise Server
• High-speed faspTM desktop transfer clients • High-speed faspTM transport of large data sets
• Easy to use drag and drop user interface and built in CLI • Enterprise-wide high-volume content ingest and distribution
• Transfer initiation and automated queuing and scheduling • Comprehensive automation and scheduling
• Advanced bandwidth utilization management • Advanced user and server management
• Graphically-adjustable transfer speed and priority • Complete monitoring and reporting
• Built in email notification and delivery confirmation • High-availability configuration (active/active or active/passive)

Connect Connect Server
• Install-on-demand Web browser plug-in • Incorporates all the features of the Enterprise Server
• High-performance fasp™ transfers directly from browsers • Web portal for high-speed upload / download of files,
• Built-in transfer monitor for visual, on-the-fly transfer rate control directories, and multi-item transfers
and monitoring • Rich set of APIs, can be integrated / embedded in 3 rd party
• HTTP fallback mode for highly-restrictive network environments web applications
• Complete Javascript API for embedding in custom Web portals • Supports Aspera Connect browser plug-in with automatic
download and installation

Mobile Sync
• High-speed fasp-AIR™ uploads from an iOS device over • High speed faspTM synchronization of remote files and
wireless networks, including cellular and Wi-Fi directories
• Easy-to-use interface that directly accesses the picture and • Highly scalable – supports today’s extremely large data sets
video library, as well as the iPhone built-in camera • One-to-one, one-to-many, and full-mesh synchronization
• Real-time transfer progress and performance display • One time, scheduled, or continuous in both push and pull
mode

Aspera Collaborate

Global person-to-person and project-based exchange and collaboration of files and directories, of any size,
over any distance, over any network.

faspexTM Server
Cargo
• High-performance fasp™ transport
• Automatic downloads of faspexTM packages from Aspera
• Person-to-person and project-based file-exchange
faspexTM servers
• Easy-to-use web-based interface
• Supports secure embargoes on content access with built-in
• Integrated email notification
encryption
• Advanced policy-based encryption
• Easy GUI controls for automatic decryption of all content in
• Drop Boxes for ad-hoc submission and distribution
a downloaded package
• Fully configurable custom Meta data support
• Configurable download target location per faspex server
• Supports Cargo for automatic package download and
and account
distribution
• Support for a configurable number of concurrent downloads
• Enterprise-scale user management and access control
and queuing
• Unlimited scale-out with multi-server relay, clustering, and
• Real-time transfer rate control and monitoring Pause, cancel
HA support
and resume functionality with automatic retry
• Supports all LDAP directory services for import,
synchronization and direct authentication
• Broad Platform support
Outlook Plugin (Coming soon)
• High-speed transfer of large file attachments seamlessly
faspexTM Multi-Server / HA from within Outlook
• Automated bi-directional relays between sites and multiple • Leverages the strength of the Aspera fasp™ transport
servers protocol and existing faspex™ software and infrastructure
• 3-tier architecture with support for clustering and high
availability

Aspera Design: Developer Network & SDK

A complete set of SDKs provides developers with guides, reference information, and sample code to assist them with integrating Aspera
technology into their own applications. Aspera fasp™ technology can be used in desktop, network-based, and web applications in place
of FTP, HTTP, or custom TCP-based copy protocols.

Aspera Transfer APIs Aspera Mobile APIs

Aspera Web Services Android SDK
A SOAP based web service API that allows initiation, Aspera Android SDK provides a Java API to transfer files using
monitoring and controlling of fasp based file transfers. fasp-AIR™.

Aspera Web iPhone SDK
Javascript API exposed by Aspera Connect client. It allows Aspera iPhone SDK provides an Objective C API to transfer files
integration of fasp based file transfers into web using fasp-AIR.
applications.

fasp Manager
A class library that allows intiations, monitoring and Aspera Application APIs
controlling of fasp based file transfers.
faspex™ Web API
Aspera Multicast SDK The Aspera faspex Web API provides a set of services that enables
A Java class library that allows initiation and management users to create and receive digital deliveries via a Web interface, while
of IP multicast based data transmissions using Aspera taking advantage of fasp high-speed transfer technology
fasp-MC™.

Other Information

Supporting Tools and Libraries General Reference
Supporting tools and libraries let you perform other common Reference on error codes, log file locations, configuration files and
tasks surrounding file transfers. more.

Why AWS for Media Production & Distribution?

• Content Creation
– Compute Intensive: EC2 (10’s, 100’s, 1000’s of CPUs)
• Transcoding, encoding, watermarking, video editing
• Rendering & HPC applications

• Mission Critical Storage and Distribution
– Long-term archive & backup
– Near-line storage for compute
– B2B/B2C media ingest & distribution

• Monetization & Play Out
– Release, project and event specific marketing & social media
– Brand awareness & franchise continuity
– CDN and Delivery
• AWS Cloud Front

Big-Data Cloud Transfer Bottlenecks

Characterizing & Understanding
Object Storage for Big-Data

Two Major Bottlenecks: WAN Transfer & Local HTTP I/O

1st Bottleneck - WAN 2nd Bottleneck – Data Center
Transfers over the WAN are TCP “Last-foot” local transfers from EC2 to
based (FTP, SCP, HTTP etc) S3 can use multiple HTTP connections

Server S3
(EC2)
WAN

1st Bottleneck: WAN Transfers

#1 WAN Transfer: Local machine to EC2 Effective throughput
• Single HTTP transfer
• Typical internet conditions 0.5 to 5 Mbps
 50-250ms latency & 0.1- 3% packet loss

• 15 parallel http streams 7 to 75 Mbps

• Aspera fasp transfer Up to 700 Mbps

WAN

fasp – Secure, Reliable & Line-Rate WAN Speed

• Secure user/endpoint authentication

• AES-128 encryption in transit & at rest

• Packet-level data integrity verification

• Automatic resume of partial or failed transfers

• Any storage i/o (NAS, SAN, DAS, Object, etc)

• Any platform (Windows, Linux, MAC, UNIX, iOS, Android)

• Any method (CLI, Desktop, Mobile, Web, Embedded)

2nd Bottleneck: Data Center/ Local HTTP I/O

2nd Bottleneck – Data Center
“Last-foot” local transfers from Server
(EC2) to Storage (S3) using
one/many HTTP connections

Server S3
(EC2)

Some History - AWS S3: 762 Billion objects and counting

Now 762 Billion.

• This represents year-over-year growth of 192%

• At 449 Billion objects (in July 2011) this was:
– 1,440 objects for every resident of the US

– 64 objects for each person on Planet Earth

– ~ the number of stars in the Milky Way

http://aws.typepad.com/aws/2012/01/amazon-s3-growth-for-2011-now-762-billion-objects.html

2006-2009 Big-Data & AWS

March 2006: Computer
Music / Audio
Modeling
– AWS Launches the S3 object storage system
June 2009:
– AWS announces physical import/export for AWS
September 2009:
– Aspera launches Aspera OnDemand for AWS Video & Photos &
Graphics Imaging

Genetic
Sequencing PDFs

Sept 2009: Aspera Launches On-Demand for AWS

• On-Demand: powered by Aspera’s patented fasp™ technology

– Next-generation transport protocol for digital media

– Eliminates the latency & packet loss bottlenecks of TCP

– Reliable & secure asset delivery system

– Replaces FTP, HTTP, NFS, CIFS, tape and disks

– Seamless integration w/ all Aspera Clients and Console Management

• Aspera On-Demand lowered the network barrier to AWS adoption:

– Solved Bottleneck #1

– But didn’t transfer data directly to/from S3 – Bottleneck #2

Aspera On-Demand Version 1 – No S3 Support…

Aspera On-Demand v.1 did not read/write data to S3

fasp
Aspera Connect Aspera S3 EC2
Browser Plugin On-Demand
Server

EC2
Aspera Client
WAN

HTTP(s) I/O only – Object Store - Key/Value Pair
Global availability – 11 9’s durability
Aspera Mobile

Local
HD

Not persistent
Elastic Block Store – EBS (NAS)
Regional only – Limited to 1TB blocks – single mount

2011: Big-Data REALLY Meets AWS…
Dec 2010: AWS Announces Major S3 Upgrade Computer
Music / Audio
Modeling
– S3 object size increased
• 5GB to 5TB
• AWS introduces multipart HTTP uploader
• API’s available in Java, .NET, PHP & REST
Fantastic… but now what? Video & Photos &
Graphics Imaging
• Still HTTP over the WAN (SLOW)
• Still have to “glue” any fasp high speed transfer
to S3 I/O in custom s/w – big speed bump!
• Find an expert s/w team
Genetic
• Build upon the multipart API Sequencing PDFs

• Concurrently stream data to S3
• Integrate into operations

S3 & Big-Data: API/ Commercial Application Options

Multi-Part Uploader API (from AWS) Commercial Tools

Step Action
1 Initiate multipart upload by providing your AWS credentials

2 Provide required bucket name and key name

3 Save the upload ID for each subsequent multipart upload operation

Upload parts providing part upload information (upload ID, bucket
4
name, part number)

5 Save the responses (ETag value and the part number)

6 Repeat tasks 4 and 5 for each part of your object

7 Execute a final call to complete the multipart upload
S3 Multi-part upload
HTTP

!"#

What are the Challenges of Storing Big Files in the Cloud?

• Designed as scalable distributed object stores
– Target applications require simple read/write operations of binary "blobs”, indexed by a single primary key
– Should work well for storing large numbers of media files, compared to traditional file systems

• BUT
– “Blob" sizes are small (<64 MB) => large media files must be “chunked”
– Data I/O use the standard HTTP protocol – VERY SLOW at distance
– API for managing data requires a team of experts

• M&E/ Big-Data services require high-speed software bridge over the WAN
– Large files to be moved at full bandwidth capacity w/ global access
– Must overcome the WAN and the I/O bottleneck
– Must allow for writing media files of any size
– Must be transparent to the end user uploading / downloading (GUI, command line, browser, etc.)

Solving the Big-Data i/o at Scale

With the help of AWS, Aspera did a full characterization of AWS S3 i/o:

• Upload/Download performance vs. thread count

• Upload/Download performance vs. chunk size

• 24hr upload stability w/ fixed thread size

• 24hr download stability w/ fixed chunk

• Upload/Download performance vs. duration

• DNS lookup performance

• Performance w/ concurrent access to single S3 bucket

• Performance w/ max connections per host

The Result? Aspera On-Demand S3-Direct
HTTP S3
Parts API
Aspera Connect
Browser Plugin

S3 S3
Aspera Server/ Client Optimized S3 i/o

Aspera
On-Demand
Aspera Mobile Server
fasp S3
Server RAM
WAN fasp-S3
Gateway
EC2

Aspera On-Demand S3-direct:
• Full client-side r/w of S3
• Synchronous transfer from Client to S3 (via EC2 Aspera On-Demand)
• Real-time optimization of HTTP threads
• Real-time optimization of chunk size Aspera fasp data session

Overcoming Both Bottlenecks - Transferring Data to S3 over WAN

#1 - Transfer Data to EC2 over WAN Effective throughput
• http transfer over WAN (single stream)
• Typical internet conditions 0.5 to 5 Mbps
• 50-250ms latency & 0.1- 3% packet loss
• 15 parallel http streams 7.5 to 100 Mbps
• Aspera fasp transfer over WAN to EC2 up to 700 Mbps

#2 - Transfer Data from EC2 to S3 Effective throughput
• Standard single stream http 20 to 100 Mbps
• Aspera S3 Proxy
up to 700 Mbps
• With parallel I/O http streams

fasp™ 45 Mbps 100 Mbps 200 Mbps 1 Gbps 5 Gbps 10Gbps
1 GB 3.2 min 1.4 min 42 sec 8.4 sec 1.6 sec 0,8 sec
10 GB 32 min 14 min 7 min 1.4 min 16 mins 8.2 sec
100 GB 5.3 hrs 2.3 hrs 1.2 hrs 14 min 2.7 mins 82 sec
1TB 2.1 days 23 hrs 11.7 hrs 2.3 hrs 28 mins 14 mins

Aspera On-Demand S3-direct

EC2
S3

Aspera Connect Aspera Client Aspera Mobile Aspera Console
Browser Plugin &/or Server (Command/ Control/
Report)

Aspera
On- RD
Deman
Aspera
Aspera d
On-
On- Server
Demand
S
Demand
Aspera
Server
EC2
HTTP
Server
EC2
On-Demand
Server EC2
(FTP/ HTTP

fasp
replacement)
S3
S3
fasp

Persistent Data
WAN Faspex
Local On-Demand
HD (Collaboration)

Elastic Block Store – EBS (NAS)
EBS

RDS – MySQL Relational Database Service

Big-Data: Accessed & Delivered by Aspera

For more information email:
ondemand@asperasoft.com

Aspera on demand for AWS (S3 inc) overview

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Aspera on demand for AWS (S3 inc) overview

Similar to Aspera on demand for AWS (S3 inc) overview (20)

Recently uploaded

Recently uploaded (20)

Aspera on demand for AWS (S3 inc) overview

Editor's Notes