span8
span4
When using FME Cloud,one of the biggest hurdles customers face is moving their datasets into the cloud—particularly if they are large.将数据移动到云端可以分为三组,每组都有不同的问题,and therefore a different recommended approaches for loading data:
Since FME Cloud runs on AWS,我们看到的大多数场景都涉及将数据加载到AmazonWeb服务(AWS)中,but these techniques also apply to loading data into other cloud platforms such as Microsoft Azure.
这是最简单的情况,因为网络带宽不会受到限制,what defines small really depends on your available bandwidth.要频繁上传小数据集,您可以利用AWS和第三方提供的许多工具。The following solutions all use HTTP,这就足够了,除非你的互联网连接真的不可靠。
将数据加载到AWS S3中
将数据加载到RDS
A database running on RDS has exactly the same interface as a database running on-premises.That means you can use standard tools to load data in.
Loading data into S3 and EBS volumes
在将大型数据集加载到云中时,网络带宽通常是一个限制。For one time bulk uploads you can use theAWS Import/Export Snowballservice or theAzure提供的类似服务.将数据加载到SSD磁盘并将其发布到Amazon,then they load your data into a nominated S3 bucket or EBS mount.如果您计划进行仅更改的更新,这是进行批量上载的极好方法。
如果通过网络加载数据需要7天或更长时间,一定要考虑使用AWS导入/导出。First,它具有成本效益,as you don't have to pay for bandwidth costs,装卸费和每小时2.49美元。Second,它是安全的:你可以使用密码和软件加密来确保你的数据在传输过程中是安全的。Finally,your data is guaranteed to load within 1 business day of receiving it,so it is a relatively fast way to load large datasets.
将数据加载到RDS
A database running on RDS has exactly the same interface as a database running on-premises.That means you can use standard tools to load data in.
这种相对常见的情况是这三种情况中最棘手的。What makes it tricky is the frequency,high frequency means you really have to use the network as the AWS import/export is too slow.
To upload large volumes of data the standard tools AWS can be too slow—even if you have a fast internet connection—as they all rely on HTTP.There is overhead with HTTP because it relies on the TCP protocol,它并不是专为跨广域网移动大型数据集而设计的。
加速文件传输解决方案已经进入市场,利用UDP,claiming they can facilitate much greater throughput by using more of your available bandwidth as they are less affected by network overhead.Several accelerated file transfer solutions exist,and I ran a series of benchmarking tests and found that the the overhead for HTTP wasn't as big as an issue as I thought.Result are available in thisblog post.
The greatest benefit I identified was reliability,并将文件上传转化为容错组件。Often we design complicated fault-tolerant architectures in the cloud,利用所有的AWS来确保我们有一个稳定的可靠的应用。然而,such a design is only as strong as the weakest link.如果您依赖上传到云端的数据来触发工作流,亚搏在线我敢打赌,这可能是最薄弱的环节。If uploading files is an integral part of your 亚搏在线workflow,我建议您看看商业加速文件传输解决方案。
Working with Amazon S3 and FME
Using and configuring S3 with FME Cloud
Using and configuring SQS with FME Cloud
Getting Started with FME Cloud: Events
How to keep the OS of your FME Cloud instance up to date
Tutorial: Getting Started with FME Cloud
FME Cloud: How to speed up your 亚搏在线workflows with the temporary disk
© 2019 亚搏在线Safe Software Inc |Legal