Problem with parallelized LaTeX runs after FNDB refresh
Patrick Lainer <[email protected]>
| Newsgroups | gmane.comp.tex.miktex |
|---|---|
| Message-ID | <[email protected]> |
Dear MiKTeX users,
I have come across a peculiar problem with multithreading: right after
refreshing the FNDB and rebuilding formats, calls to TeX & friends
fail when executed in parallel because each process attempts to
refresh .fndb files residing in the UserData directory tree (the
output of initexmf --report is attached as report.txt) at the same
time. Subsequent runs are not affected. Here is an almost minimal
working example Windows batch file to reproduce the problem:
@echo off
call :timestamp clean up from the previous run
if exist "test*.*" del "test*.*"
call :timestamp refresh MiKTeX
initexmf --admin --verbose --update-fndb --dump=lualatex 2>nul
initexmf --update-fndb --verbose --dump=lualatex 2>nul
luaotfload-tool -vvv --update
call :timestamp generate test batch file
echo @echo off > test.bat
setlocal EnableDelayedExpansion
for /L %%i in (1, 1, %NUMBER_OF_PROCESSORS%) do (
:: affinity for a processor core has to be given in hexadecimal
call cmd /c exit /b %%i
set hex=!=exitcode!
echo start /max /affinity !hex! cmd /k lualatex
--interaction=nonstopmode --jobname=test%%i
\documentclass{article}\begin{document}Test\end{document} >> test.bat
)
endlocal
call :timestamp first round of tests after refreshing MiKTeX
call test.bat
pause
call :timestamp second round of tests after partially failed first round
call test.bat
goto :EOF
:timestamp
:: log time for comparison with log files
echo.
echo -- %DATE% %TIME% -- %* --
echo.
:EOF
After the first round of tests, new shells pop up with some of them
showing an error. Apparently it's a statistical process -- in my case,
0 to 5 out of 8 will fail, so maybe you have to run the script more
than once. The error messages point to the log files for more details,
so I attached the relevant sections of lualatex.log and initexmf.log.
As said before, no errors appear in the second round without an
intervening database refresh. A sample output of my script
corresponding to the log files is attached as output.txt.
Note that luaotfload-tool is not relevant and only shows up for the
sake of completeness. I first noticed that it sometimes calls initexmf
when no FNDB refresh was made beforehand. I also inspected the access
to files with Process Monitor from Windows Sysinternals and noticed
that on the first try each process attempts to lock the .fndb file for
writing at the same time.
I'm not entirely sure about the source code, but the function
Application::Init in
https://github.com/MiKTeX/miktex/blob/master/Libraries/MiKTeX/App/app.cpp#L139
looks like every MiKTeX executable checks the FNDB and refreshes it
when necessary. It appears that after my manual call to initexmf the
executable called next refreshes the FNDB again anyway.
I've had a lot of trial and error to arrive at the above example and
just now found a workaround: a "sacrificial" call to lualatex after
the refresh and before parallelization will repair the FNDB without
the conflict inherent in parallelization. Please correct me if I made
a mistake; otherwise it would be interesting to know why the call to
initexmf doesn't work as expected.
Best regards,
Patrick
--
Basisgruppe NAWI Physik (http://physik.htu.tugraz.at)
Referat für Bildungspolitik (http://bipol.htu.tugraz.at)
Curriculakommission Bachelor/Master ([email protected])
Technische Universität Graz
------------------------------------------------------------------------------
_______________________________________________
Q: How can I leave the mailing list?
A: See http://docs.miktex.org/faq/support.html#leavingml
output.txt
(text/plain, 3.6 KB)
-- 13.09.2016 15:27:49,70 -- clean up from the previous run --
-- 13.09.2016 15:27:49,71 -- refresh MiKTeX --
Creating the lualatex format file...
Running miktex-luatex.exe...
Skipping user root directory (C:\Users\patrick.lainer\AppData\Roaming\MiKTeX\2.9)...
Skipping user root directory (C:\Users\patrick.lainer\AppData\Local\MiKTeX\2.9)...
Creating fndb for common root directory (C:\ProgramData\MiKTeX\2.9)...
Creating fndb for common root directory ("C:\Program Files (x86)\MiKTeX 2.9")...
Creating the lualatex format file...
Running miktex-luatex.exe...
Creating fndb for user root directory (C:\Users\patrick.lainer\AppData\Roaming\MiKTeX\2.9)...
Creating fndb for user root directory (C:\Users\patrick.lainer\AppData\Local\MiKTeX\2.9)...
Skipping common root directory (C:\ProgramData\MiKTeX\2.9)...
Skipping common root directory ("C:\Program Files (x86)\MiKTeX 2.9")...
luaotfload | util : Setting the log level to 3.
luaotfload | util : Task completed successfully.
luaotfload | db : Updating the font names database.
luaotfload | db : Font names database loaded from C:/Users/patrick.lainer/AppData/Local/MiKTeX/2.9/luatex-cache/generic/names/luaotfload-names.luc
luaotfload | db : Loading took 16 ms.
luaotfload | db : Blacklisted file "spltfgbd.ttf" via "C:/Program Files (x86)/MiKTeX 2.9/tex/luatex/luaotfload/luaotfload-blacklist.cnf".
luaotfload | db : Blacklisted file "spltfgbi.ttf" via "C:/Program Files (x86)/MiKTeX 2.9/tex/luatex/luaotfload/luaotfload-blacklist.cnf".
luaotfload | db : Blacklisted file "spltfgit.ttf" via "C:/Program Files (x86)/MiKTeX 2.9/tex/luatex/luaotfload/luaotfload-blacklist.cnf".
luaotfload | db : Blacklisted file "spltfgrg.ttf" via "C:/Program Files (x86)/MiKTeX 2.9/tex/luatex/luaotfload/luaotfload-blacklist.cnf".
luaotfload | db : Blacklisted file "spltfgbd.ttf" via "C:/Program Files (x86)/MiKTeX 2.9/tex/luatex/luaotfload/luaotfload-blacklist.cnf".
luaotfload | db : Blacklisted file "spltfgbi.ttf" via "C:/Program Files (x86)/MiKTeX 2.9/tex/luatex/luaotfload/luaotfload-blacklist.cnf".
luaotfload | db : Blacklisted file "spltfgit.ttf" via "C:/Program Files (x86)/MiKTeX 2.9/tex/luatex/luaotfload/luaotfload-blacklist.cnf".
luaotfload | db : Blacklisted file "spltfgrg.ttf" via "C:/Program Files (x86)/MiKTeX 2.9/tex/luatex/luaotfload/luaotfload-blacklist.cnf".
luaotfload | db : Blacklisting 8 files and directories.
luaotfload | db : Whitelisting 0 files.
luaotfload | db : Scanning TEXMF and $OSFONTDIR for fonts...
luaotfload | db : Path ÔÇ£.ÔÇØ matches $PWD (ÔÇ£d:/downloads/mweÔÇØ), skipping.
luaotfload | db : Path ÔÇ£.ÔÇØ matches $PWD (ÔÇ£d:/downloads/mweÔÇØ), skipping.
luaotfload | db : Initiating scan of 148 directories.
luaotfload | db : Collected 2901 files.
luaotfload | db : Scanning system fonts...
luaotfload | db : Searching in static system directories...
luaotfload | db : Collected 1096 files.
luaotfload | db : Scanning 3997 collected font files ...
luaotfload | db : Found 3997 font files; 0 new, 0 stale entries.
luaotfload | db : Creating filename map.
luaotfload | db : Analyzing families.
luaotfload | db : Analyzing shapes, weights, and styles.
luaotfload | db : Ordering design sizes.
luaotfload | db : Rebuilt in 24024 ms.
luaotfload | db : No new or removed fonts, skip saving to disk.
luaotfload | cache : Lookup cache saved.
luaotfload | cache : Lookup cache emptied.
luaotfload | db : Fonts in the database: 2365
luaotfload | util : Task completed successfully.
-- 13.09.2016 15:28:48,12 -- generate test batch file --
-- 13.09.2016 15:28:48,30 -- first round of tests after refreshing MiKTeX --
Drücken Sie eine beliebige Taste . . .
-- 13.09.2016 15:29:15,83 -- second round of tests after partially failed first round --
report.txt
(text/plain, 666 B)
MiKTeX: 2.9 Invokers: non-existing/explorer/cmd SharedSetup: yes SystemAdmin: yes PowerUser: no OS: Windows 7 Enterprise, 64-bit, Service Pack 1, build 7601 Root #0: C:\Users\patrick.lainer\AppData\Roaming\MiKTeX\2.9 Root #1: C:\Users\patrick.lainer\AppData\Local\MiKTeX\2.9 Root #2: C:\ProgramData\MiKTeX\2.9 Root #3: C:\Program Files (x86)\MiKTeX 2.9 UserInstall: C:\Users\patrick.lainer\AppData\Roaming\MiKTeX\2.9 UserData: C:\Users\patrick.lainer\AppData\Local\MiKTeX\2.9 UserConfig: C:\Users\patrick.lainer\AppData\Roaming\MiKTeX\2.9 CommonInstall: C:\Program Files (x86)\MiKTeX 2.9 CommonData: C:\ProgramData\MiKTeX\2.9 CommonConfig: C:\ProgramData\MiKTeX\2.9