Re: Pycon sprint hack: storing pandas data frames in ZODB blobs

Hi, this sounds very interesting but I do have a question.  I store pandas 
dataframes in blobs under zeo/zodb by pickling as follows below (and this 
works great).  I realize that this is not terribly efficient but is there 
some other 'gotchas' with this approach that your pheather work is 
attempting to address or is your work more of a optimization effort?  
Thanks very much, pjfd.
    def save_df_as_blob(self,my_dataframe,key):
        self.blobs[key] = Blob()
        s = pickle.dumps(my_dataframe)
        f = self.blobs[key].open('w')
        f.write(s)
        f.close()




On Saturday, May 27, 2017 at 9:42:34 AM UTC-4, Jim Fulton wrote:
>
>
> IMO, the main benefit of sprints is social. To that end, at PyCon this 
> year I sought to collaborate with someone in the data science community. I 
> implemented a little package, *pheather*, for storing pandas data frames 
> in ZODB blobs:
>
>   https://github.com/jimfulton/pheather
>
> This was a quick hack, although not as quick as I hoped.  It currently has 
> no tests.  I was able to store data sets both in FileStorage and 
> RelStorage/Postgres.  Alas, the person I spoke with this who was going to 
> play with this left before I was ready.  Something else came up that I 
> needed to focus on so I didn't show this to anyone else.
>
> I'm not positive, but I think there's some potential for ZODB to be of use 
> to data scientists to share data.  I think the data scientists at the last 
> company I was at could have benefited from this.
>
> Maybe someone would like to take this further.
>
> Jim
>
> -- 
> Jim Fulton
> http://jimfulton.info
>

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.