span8
span4
span8
span4
I am trying to read from two large csv files with one FeatureReader for each file.Each FeatureReader is connected to a separate FeatureWriter to write to a Postgres database.I experience the problem that all data is cached first and then delivered to the writer, which will eventually exceed my memory.Since there are no transformers in between I do not now how to solve this issue.
I experienced this problems in the past when I used ordinary writers, but never with a FeatureWriter, so I guess it has something to do with the usage of FeatureReader.
That sounds more like a limitation of your system resources than a FME issue.
Are you by any chance running the translation with feature caching on?(that can be quite memory intensive)
I would not say that this is a resource limitation.The files are in the magnitude of some terabyte, something the database servers can handle, but not my local machine.The usual workflow would normally be like: take X rows from file and put them into the database.This would not take large amounts of memory, both ram and physical.
FME tries to read all data and then begins to put it into the databases, which is quite slow and takes large amounts of memory.
Feature caching is disabled, that was my first guess to.
Just to clarify by "your system resources" I did actually mean your local machine resources.
Could it be that the FME TEMP is not configured for optimal efficiency?https://knowledge.safe.com/articles/176/fme-temp-environment-variable.html
My local machine is not configured to handle terabytes of temporary files, that is correct.
如果我想用外部的编程语言如Python做的工作,我可以做我想做的恒定RAM和物理内存,但由于读写管道具有总是相同的尺寸。我不知道这是否会在FME完成,以及。我没有看到存储大量数据,以便把它们写在将来,当数据可以被写入并立即丢弃的地步。
© 2019 Safe Software Inc |Legal