Re: Large files on Windows?

Hi Neil,
I just discovered that the issue is with the right click options in the 
SciTE for Windows I am using.
If I open SciTE and then I go to "File, Open" and I select my merged fasta 
file, I see a progress bar and then the file is there (scroll bar at the 
top, like you said), no NUL values and all is fine, meaning that also 
scrolling through the large fasta file is really a nice experience.
If I right click (like I always do) on my merged fasta file and I select 
either "Edit with SciTE in New Tab" or "Edit with SciTE in New Window" I 
don't get the progress bar and I get what I detailed below (NUL values) and 
also the scrolling experience is really slow.
Hope this helps.
Nedo

Il giorno lunedì 20 novembre 2023 alle 09:09:57 UTC Nedo Papanti ha scritto:

> Hi Neil,
>
> Thank you for your prompt feedback.
>
> - The two (randomly chosen for testing purposes) fasta files I used are 
> public and are these two (two locations each so that people can choose the 
> fastest location for them; they are regularly updated but the same versions 
> I used for my original email will be still there for a couple of months or 
> so):
>
> https://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/uniprot_sprot_varsplic.fasta.gz
>
> https://ftp.ebi.ac.uk/pub/databases/uniprot/current_release/knowledgebase/complete/uniprot_sprot_varsplic.fasta.gz
>
> https://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/uniprot_sprot.fasta.gz
>
> https://ftp.ebi.ac.uk/pub/databases/uniprot/current_release/knowledgebase/complete/uniprot_sprot.fasta.gz
> SciTE (as well as other programs) can open/view/edit these two file 
> without issues as far as my current experience is concerned and I didn't 
> stumble in any NUL values in these two files over the many years I've been 
> using them with SciTE and other programs.
> Unfortunately I won't be able to count the NUL values any time soon 
> (despite you helpful and clear details) so I prefer to give full details of 
> what I did hoping that someone with experience could give advise.
> - In order to create a test file larger than 2,000,000,000 bytes (so to be 
> able to test for the first time SciTE with large files) what I did is 
> pasted together these two fasta files on the Windows command prompt with:
> copy /b uniprot_sprot_varsplic.fasta+uniprot_sprot_varsplic.fasta UP.fasta
> This UP.fasta file can be opened/viewed/edited by SciTE and other programs 
> without issues and I didn't stumble in any NUL values in this file so I 
> guess the "copy /b" command didn't do any harm in that sense (I've been 
> doing this type of merging of this type of files for years and years and 
> never had any sort of issue whatsoever with the upstream use I made of 
> them).
> Next I created eight copies of the same UP.fasta and I merged them 
> together again with "copy /b" to have the 2,510,747,416 bytes test file I 
> mentioned in my original email.
> - The RAM of the machine I used is 32 GB
> - Some of the fasta headers (a fasta header is what comes after the ">" 
> character up to the first line break) can be long; quite a lot more than 
> the 60-character line break in the sequence area (the sequence area is the 
> text in between two ">" characters and excluding the fasta headers lines)
>
> With this info it should be possible to try reproducing my issue (Windows 
> and SciTE for Windows details in my original email).
>
> Hope this additional info helps in trying to figure out if there's space 
> for improvement in my usage of SciTE with these type of large files.
>
> Thanks again
>
> Best Regards
>
> Emanuele
>
> Il giorno sabato 18 novembre 2023 alle 10:01:44 UTC Neil Hodgson ha 
> scritto:
>
>> Hi Nedo,
>>
>> When I try to open a file which is large 2,510,747,416 bytes (it is an 
>> aminoacidic sequence database in fasta format)
>>
>>
>> OK, looked up fasta and it should have short (< 200 byte) lines. Very 
>> long lines can cause performance problems. Extremely long lines may cause 
>> failures.
>>
>> If you own the file or have permission to make it available, it may help 
>> to publish it, preferrably compressed.
>>
>> , SciTE warns me that the file is larger than the 2,000,000,000 bytes 
>> limit set in the properties and I am asked if I still want to open it; when 
>> I click on "Yes" I wait for a while and then the SciTE window fills up with 
>> "NUL" texts and
>>
>>
>>  The most likely reason for NUL blobs appearing is that the file contains 
>> NUL bytes.
>>
>> SciTE uses RAM to store the entire contents of files and also needs more 
>> memory to index lines, so may not work well on this file if your computer 
>> contains less than 4GB of RAM.
>>  
>>
>> the vertical scroll bar is at the bottom.
>>
>>
>> That is unusual: SciTE normally shows the start of newly opened files.
>>  
>>
>> If I move the scroll bar up above a certain point I start seeing the 
>> correct text I am expecting but then if I scroll back down, at a certain 
>> point, the "NUL" texts fills the SciTE window again.
>>
>>
>> That may mean that there are blocks of NUL bytes at the file end. 
>>
>> I am aware that there are other options to deal with big files of this 
>> type but, despite this, is there anything that I can do to improve this 
>> with SciTE?
>>
>>
>> First determine if there is a problem: count the number of NUL bytes in 
>> the file with a script. Something like this piece of Python with fileName 
>> set to the path to your file.
>>
>> fileName = "../bin/SciTE.exe"
>> import pathlib
>> bytes = pathlib.Path(fileName).read_bytes()
>> print(f"{fileName} contains {bytes.count(b'\0')} NULs in {len(bytes)} 
>> bytes")
>>  
>>
>> I have searched this Google group with "file.size.large" and I read also 
>> the sentence from Neil Hodgson saying "An undocumented file.size.large 
>> property was added to SciTE that allowed loading files larger than 2GB. 
>> However, SciTE uses 32-bit integers for file positions so many features 
>> wouldn’t work past the 2GB point" but I am not able to say with certainty 
>> if there is something that can be done for my case.
>>
>>
>> The 64-bit version of SciTE has used 64-bit positions for over 3 years 
>> now and can load 2.5 GB files - I just loaded a 4GB file and it worked 
>> correctly.
>>
>> Neil
>>
>

-- 
You received this message because you are subscribed to the Google Groups "scite-interest" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web visit https://groups.google.com/d/msgid/scite-interest/3fc7704e-19b9-4b41-9b00-036bfa6ab19en%40googlegroups.com.