Re: Pycon sprint hack: storing pandas data frames in ZODB blobs
Jim Fulton <[email protected]>
| Newsgroups | gmane.comp.web.zope.zodb |
|---|---|
| Message-ID | <CAPDm-Fgz7vfYa4iasSj7cwSn-SG0De+fA0qx8EVXFHpP3rtuXw@mail.gmail.com> |
On Thu, Jun 8, 2017 at 6:47 PM, PJF Dixon <[email protected]> wrote: > Hi, this sounds very interesting but I do have a question. I store pandas > dataframes in blobs under zeo/zodb by pickling as follows below (and this > works great). I realize that this is not terribly efficient but is there > some other 'gotchas' with this approach that your pheather work is > attempting to address or is your work more of a optimization effort? > Thanks very much, pjfd. > def save_df_as_blob(self,my_dataframe,key): > self.blobs[key] = Blob() > s = pickle.dumps(my_dataframe) > f = self.blobs[key].open('w') > f.write(s) > f.close() > IIRC the pandas docs suggest that pickles may break due to data-frame implementation changes. <shrug> I assumed that feather would be faster and more stable. Perhaps you can compare speed? :) BTW, another interesting and wildly more ambitious project along these lines is wendelin, https://github.com/Nexedi/wendelin.core. Jim > > > > > > On Saturday, May 27, 2017 at 9:42:34 AM UTC-4, Jim Fulton wrote: >> >> >> IMO, the main benefit of sprints is social. To that end, at PyCon this >> year I sought to collaborate with someone in the data science community. I >> implemented a little package, *pheather*, for storing pandas data frames >> in ZODB blobs: >> >> https://github.com/jimfulton/pheather >> >> This was a quick hack, although not as quick as I hoped. It currently >> has no tests. I was able to store data sets both in FileStorage and >> RelStorage/Postgres. Alas, the person I spoke with this who was going to >> play with this left before I was ready. Something else came up that I >> needed to focus on so I didn't show this to anyone else. >> >> I'm not positive, but I think there's some potential for ZODB to be of >> use to data scientists to share data. I think the data scientists at the >> last company I was at could have benefited from this. >> >> Maybe someone would like to take this further. >> >> Jim >> >> -- >> Jim Fulton >> http://jimfulton.info >> > -- Jim Fulton http://jimfulton.info -- You received this message because you are subscribed to the Google Groups "zodb" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.