-
|
Another question important to time series-related problems is how much history you are getting into dataframe transformations (those that you decorate with) I'm asking this since with a continuous time series, given that we have history for a few years on e.g. minute or second resolution, that can easily mean millions of rows of data. Does the df_transformations somehow compute the transformation only for exactly the relevant time periods, or will extra history be pulled in? And, while too much history seems like it could have performance penalties, one might also sometimes make sure to include a certain amount of history in order to define time-lagged features, such as the value of a specific feature 24 hours, 1 week, 1 month or 1 year back. Thus, is there some way to control how much history is included? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
By default, FF will use all of the existing data. I wouldn't expect a few million of rows to take very long though, less than a couple mins. Another technique we've seen is to create a transformation on the main dataset that filters all rows before a specific point of time then run your transformations/features/etc. on top of that table. |
Beta Was this translation helpful? Give feedback.
By default, FF will use all of the existing data. I wouldn't expect a few million of rows to take very long though, less than a couple mins. Another technique we've seen is to create a transformation on the main dataset that filters all rows before a specific point of time then run your transformations/features/etc. on top of that table.