Re: Large files on Windows?

Hi Neil,

Thank you for your prompt feedback.

- The two (randomly chosen for testing purposes) fasta files I used are 
public and are these two (two locations each so that people can choose the 
fastest location for them; they are regularly updated but the same versions 
I used for my original email will be still there for a couple of months or 
so):
https://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/uniprot_sprot_varsplic.fasta.gz
https://ftp.ebi.ac.uk/pub/databases/uniprot/current_release/knowledgebase/complete/uniprot_sprot_varsplic.fasta.gz
https://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/uniprot_sprot.fasta.gz
https://ftp.ebi.ac.uk/pub/databases/uniprot/current_release/knowledgebase/complete/uniprot_sprot.fasta.gz
SciTE (as well as other programs) can open/view/edit these two file without 
issues as far as my current experience is concerned and I didn't stumble in 
any NUL values in these two files over the many years I've been using them 
with SciTE and other programs.
Unfortunately I won't be able to count the NUL values any time soon 
(despite you helpful and clear details) so I prefer to give full details of 
what I did hoping that someone with experience could give advise.
- In order to create a test file larger than 2,000,000,000 bytes (so to be 
able to test for the first time SciTE with large files) what I did is 
pasted together these two fasta files on the Windows command prompt with:
copy /b uniprot_sprot_varsplic.fasta+uniprot_sprot_varsplic.fasta UP.fasta
This UP.fasta file can be opened/viewed/edited by SciTE and other programs 
without issues and I didn't stumble in any NUL values in this file so I 
guess the "copy /b" command didn't do any harm in that sense (I've been 
doing this type of merging of this type of files for years and years and 
never had any sort of issue whatsoever with the upstream use I made of 
them).
Next I created eight copies of the same UP.fasta and I merged them together 
again with "copy /b" to have the 2,510,747,416 bytes test file I mentioned 
in my original email.
- The RAM of the machine I used is 32 GB
- Some of the fasta headers (a fasta header is what comes after the ">" 
character up to the first line break) can be long; quite a lot more than 
the 60-character line break in the sequence area (the sequence area is the 
text in between two ">" characters and excluding the fasta headers lines)

With this info it should be possible to try reproducing my issue (Windows 
and SciTE for Windows details in my original email).

Hope this additional info helps in trying to figure out if there's space 
for improvement in my usage of SciTE with these type of large files.

Thanks again

Best Regards

Emanuele

Il giorno sabato 18 novembre 2023 alle 10:01:44 UTC Neil Hodgson ha scritto:

> Hi Nedo,
>
> When I try to open a file which is large 2,510,747,416 bytes (it is an 
> aminoacidic sequence database in fasta format)
>
>
> OK, looked up fasta and it should have short (< 200 byte) lines. Very long 
> lines can cause performance problems. Extremely long lines may cause 
> failures.
>
> If you own the file or have permission to make it available, it may help 
> to publish it, preferrably compressed.
>
> , SciTE warns me that the file is larger than the 2,000,000,000 bytes 
> limit set in the properties and I am asked if I still want to open it; when 
> I click on "Yes" I wait for a while and then the SciTE window fills up with 
> "NUL" texts and
>
>
>  The most likely reason for NUL blobs appearing is that the file contains 
> NUL bytes.
>
> SciTE uses RAM to store the entire contents of files and also needs more 
> memory to index lines, so may not work well on this file if your computer 
> contains less than 4GB of RAM.
>  
>
> the vertical scroll bar is at the bottom.
>
>
> That is unusual: SciTE normally shows the start of newly opened files.
>  
>
> If I move the scroll bar up above a certain point I start seeing the 
> correct text I am expecting but then if I scroll back down, at a certain 
> point, the "NUL" texts fills the SciTE window again.
>
>
> That may mean that there are blocks of NUL bytes at the file end. 
>
> I am aware that there are other options to deal with big files of this 
> type but, despite this, is there anything that I can do to improve this 
> with SciTE?
>
>
> First determine if there is a problem: count the number of NUL bytes in 
> the file with a script. Something like this piece of Python with fileName 
> set to the path to your file.
>
> fileName = "../bin/SciTE.exe"
> import pathlib
> bytes = pathlib.Path(fileName).read_bytes()
> print(f"{fileName} contains {bytes.count(b'\0')} NULs in {len(bytes)} 
> bytes")
>  
>
> I have searched this Google group with "file.size.large" and I read also 
> the sentence from Neil Hodgson saying "An undocumented file.size.large 
> property was added to SciTE that allowed loading files larger than 2GB. 
> However, SciTE uses 32-bit integers for file positions so many features 
> wouldn’t work past the 2GB point" but I am not able to say with certainty 
> if there is something that can be done for my case.
>
>
> The 64-bit version of SciTE has used 64-bit positions for over 3 years now 
> and can load 2.5 GB files - I just loaded a 4GB file and it worked 
> correctly.
>
> Neil
>

-- 
You received this message because you are subscribed to the Google Groups "scite-interest" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web visit https://groups.google.com/d/msgid/scite-interest/fe694259-8300-4da6-8175-dd443a0c1f33n%40googlegroups.com.