span8
span4
span8
span4
As of FME 2019.0, the S3 transformers have been combined into one transformer known as the S3Connector transformer.For more information please see the documentation.
In this example, we will download a Park MapInfo TAB file from a Private Amazon AWS S3 bucket called FMETest, which is located in a folder called Parks.We will translate the data to Esri shapefile before uploading the resulting Esri shapefile back to the Amazon AWS S3 bucket FMETest where we will place it in a subfolder of the Parks folder.
Before beginning, you must have your Amazon AWS credentials 'Access Key ID', and 'Secret Access Key'.
We will begin with a blank workspace.
A Creator is added so that we can start the translation.
Next, add an S3ObjectLister.This will list all of the objects in a specific S3 bucket, so that we can then download them using an S3Downloader.
Open the S3ObjectLister parameters, and set the Access Key ID and Secret Access Key.We will be using these parameters in the S3Downloader and S3Uploader, so we'll save them as Private Parameters for easier future use.
For the S3 Object List Type, we'll select Folder, because we are interested in a specific folder within our bucket.
The Bucket Name is FMETest.
The folder Name is Parks.
We do NOT want to include subfolders.
We can run the translation with Full Inspection.Note that there is a feature for each file in the bucket.
Now we add an S3Downloader to download each file.
Open the S3Downloader parameters, and select the AWS Credentials from our previously set User Parameters.
We will download from the FMETest bucket.The Object Key is the _s3_object_key from the S3ObjectLister.
Now we set the output options.We want to actually save the files;not store their contents as attributes.
The Target Filename will be the local folder concatenated with the _s3_object_key attribute.It is VERY important to note that the S3Downloader cannot create folders or overwrite existing files.Make sure the output folder exists, and is empty.
We can now run the translation again, and confirm that the output files are created.Once you've confirmed this, be sure to delete the files.
We will now read in the downloaded MapInfo TAB file and translate it to Esri shapefile by using a FeatureReader and FeatureWriter.
The FeatureReader will read in a file for each feature it is given.We only want to use the .TAB files, so we add a StringSearcher to filter the features.
identifi FeatureReader然后读取的文件ed by the StringSearcher.
We can now use a FeatureWriter to create the output Esri shapefile.Connect the FeatureWriter to BOTH
We can now add the S3Uploader.Open the Parameters, and set the AWS Credentials.
The Data Source is Folder.We will upload the contents of c:\temp\Parks\Shape.
The Bucket Name is FMETest, and the folder is Parks.
Run the translation, and confirm that the Esri shapefile was created locally.Then check the FMETest bucket Parks folder for a new Shape folder, and check that the Esri shapefile was created.
Excellent example, especially the included .fmwt.I couldn't get the video to play from here or within YouTube but the .fmwt and written instructions were enough for success.However, it is undesirable to have to download and store the data locally as it can take some time to download (plus, one then needs to ensure folder exists and clean up after oneself after the process completes).Is there a way to go directly from the listener to the StringSearcher or FeatureReader to avoid these bottlenecks / extra processing steps?
@lovegisIf you ZIP the source data, and make the ZIP file public, you can read the source data directly by using the URL in the Reader (or FeatureReader) dataset setting.
While the .zip solution does provide a much better experience, our internal security will not allow us to store data in s3 as "public" as it can be compromised.Our internal AWS accounts have moved away from access key/secret access key as well.Are there plans at Safe to modify the s3 tools to work with IAM user accounts instead?Thanks!
Quick Tips for ArcGIS Online frequently asked questions
Geocoding with FME Using Gisgraphy
Making Weather Map Using geonames.org Web Service
Writing Incremental Updates to Cloud-Based Formats
Set up FME for Linux on Amazon EC2
Cannot connect to Amazon EC2 Instance
How do I write updates to an existing ArcGIS Online feature service?
© 2020 Safe Software Inc |Legal