Re: Memory managment on EOF
Mark Ritchie <[email protected]> Sat, 27 Aug 2005 12:20:51 -0400
| Newsgroups | gmane.comp.web.webobjects.devel,gmane.comp.web.webobjects.eof |
|---|---|
| Message-ID | <[email protected]> |
--===============0962915551== Content-Type: multipart/alternative; boundary=Apple-Mail-1-823563660 --Apple-Mail-1-823563660 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed On 25-Aug-05, at 1:13 PM, Pierce T. Wetter III wrote: > There used to be an EOUtil that could do something like this. > Does that still exists? Hi Everyone, Yes it does.. For WOF 5.3, it lives here: /System/Library/WebObjects/JavaApplications/javaeoutil.woa/javaeoutil However, as I recall, I had a bunch of problems with it. Not the least of which was that it could not handle moving some data types. As I recall, there was a problem with large blobs and it for sure doesn't handle embedded special characters. That's what killed it for me. So, I did what any of us would do... I wrote my own. :-) It copies from a source database according to a source EOModel into a target database using a target EOModel. It uses the connection dictionary in each EOModel to determine how to connect. The data is never taken higher then the EOAccess level so the problems of memory management are avoided. Data is read from the source adaptorChannel as a dictionary and written to the destination adaptor channel, one dictionary at a time. The data for each attribute is converted from the source format to the destination format (probably needs more testing here.) There are flags to control which entities are copied or conversely which ones are skipped. There's even a flag to control the batchSize so that the rollback segments don't get too large on the target database. Finally, there's some support for truncating the target tables (Oracle specific) or deleting all records so that you can script a complete copy from one database to another at regular intervals. (Use with caution as it really does delete the target database before copying over new records.) Oh and to boot, it gives details of what it's doing with a status update every 2 minutes. I've used it over the last couple of years to copy Sybase to Oracle, and Oracle to MySQL. I haven't tested it with other databases but I see no reason why it wouldn't work. I have tested it with larger data sets and it works reliably without running out of memory. For faster performance, make sure there are good, short network connections machines! Please note, it does straight copy, Entity to Entity and Attribute for Attribute. I didn't need to do any fancy mapping so I didn't add any features like that. I suppose that you could omit attribute from your source EOModel if you didn't want them copied. I'd be happy to track down the source and send it along if you want to give it a try. Please note that while it works for me, I can't guarantee that it will work for you. :-) Also, I'm leaving for Japan in 10 days so let me know soon, otherwise, I'll be back in October. Regards, Mark. __ Mark Ritchie [email protected] 416-843-5479 WebObjects, Cocoa, Mac OS X, and Mac OS X Server Diamond Lake Consulting Inc., Toronto, Ontario, Canada --Apple-Mail-1-823563660 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=ISO-8859-1 <HTML><BODY style=3D"word-wrap: break-word; -khtml-nbsp-mode: space; = -khtml-line-break: after-white-space; "><DIV><DIV>On 25-Aug-05, at 1:13 = PM, Pierce T. Wetter III wrote:</DIV><BLOCKQUOTE type=3D"cite"><P = style=3D"margin: 0.0px 0.0px 0.0px 0.0px"><FONT face=3D"Helvetica" = size=3D"3" style=3D"font: 12.0px Helvetica"><SPAN = class=3D"Apple-converted-space">=A0 </SPAN>There used to be an EOUtil = that could do something like this. Does that still = exists?</FONT></P></BLOCKQUOTE></DIV><BR><DIV>Hi Everyone,</DIV><DIV><BR = class=3D"khtml-block-placeholder"></DIV><DIV>Yes it does.. For WOF 5.3, = it lives = here:</DIV><DIV>/System/Library/WebObjects/JavaApplications/javaeoutil.woa= /javaeoutil</DIV><DIV><BR = class=3D"khtml-block-placeholder"></DIV><DIV>However, as I recall, I had = a bunch of problems with it.=A0 Not the least of which was that it could = not handle moving some data types.=A0=A0As I recall, there was a problem = with large blobs and it for sure doesn't handle embedded special = characters.=A0 That's what killed it for me.</DIV><DIV><BR = class=3D"khtml-block-placeholder"></DIV><DIV>So, I did what any of us = would do... I wrote my own. :-)</DIV><DIV><BR = class=3D"khtml-block-placeholder"></DIV><DIV>It copies from a source = database according to a source EOModel=A0 into a target database using a = target EOModel.</DIV><DIV>It uses the connection dictionary in each = EOModel to determine how to connect.</DIV><DIV>The data is never taken = higher then the EOAccess level so the problems of memory management are = avoided.</DIV><DIV>Data is read from the source adaptorChannel as a = dictionary and written to the destination adaptor channel, one = dictionary at a time.</DIV><DIV>The data for each attribute is converted = from the source format to the destination format (probably needs more = testing here.)</DIV><DIV>There are flags to control which entities are = copied or conversely which ones are skipped.</DIV><DIV>There's even a = flag to control the batchSize so that the rollback segments don't get = too large on the target database.</DIV><DIV>Finally, there's some = support for truncating the target tables (Oracle specific) or deleting = all records so that you can script a complete copy from one database to = another at regular intervals. (Use with caution as it really does delete = the target database before copying over new records.)</DIV><DIV>Oh and = to boot, it gives details of what it's doing with a status update every = 2 minutes.</DIV><DIV><BR class=3D"khtml-block-placeholder"></DIV><DIV>I've= used it over the last couple of years to copy Sybase to Oracle,=A0 and = Oracle to MySQL.</DIV><DIV>I haven't tested it with other databases but = I see no reason why it wouldn't work.</DIV><DIV>I have tested it with = larger data sets and it works reliably without running out of = memory.</DIV><DIV>For faster performance, make sure there are good, = short network connections machines!</DIV><DIV><BR = class=3D"khtml-block-placeholder"></DIV><DIV>Please note, it does = straight copy, Entity to Entity and Attribute for Attribute.</DIV><DIV>I = didn't need to do any fancy mapping so I didn't add any features like = that.</DIV><DIV>I suppose that you could omit attribute from your source = EOModel if you didn't want them copied.</DIV><DIV><BR = class=3D"khtml-block-placeholder"></DIV><DIV>I'd be happy to track down = the source and send it along if you want to give it a = try.</DIV><DIV>Please note that while it works for me, I can't guarantee = that it will work for you.=A0 :-)</DIV><DIV><BR = class=3D"khtml-block-placeholder"></DIV><DIV>Also, I'm leaving for Japan = in 10 days so let me know soon, otherwise, I'll be back in = October.</DIV><DIV><BR = class=3D"khtml-block-placeholder"></DIV><DIV>Regards,</DIV><DIV>Mark.</DIV= ><DIV>__</DIV><DIV>Mark Ritchie=A0 <A = href=3D"mailto:[email protected]">[email protected]</A>=A0 = 416-843-5479</DIV><DIV>WebObjects, Cocoa, Mac OS X, and Mac OS X = Server</DIV><DIV>Diamond Lake Consulting Inc., Toronto, Ontario, = Canada</DIV><DIV><BR class=3D"khtml-block-placeholder"></DIV><DIV = style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; "><FONT class=3D"Apple-style-span" face=3D"Courier" = size=3D"3"><SPAN class=3D"Apple-style-span" style=3D"font-size: = 13px;"></SPAN></FONT></DIV></BODY></HTML>= --Apple-Mail-1-823563660-- --===============0962915551== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ WebObjects-dev mailing list [email protected] http://www.omnigroup.com/mailman/listinfo/webobjects-dev --===============0962915551==--