Re: Solr with HDFS configuration example running in production/dev

Are you running with solr.lock.type=hdfs
?

Have you defined your DirectoryFactory - something like:

<directoryFactory name="DirectoryFactory"
         class="solr.HdfsDirectoryFactory">
         <bool name="solr.hdfs.blockcache.enabled">true</bool>
         <bool name="solr.hdfs.blockcache.global">true</bool>
         <int name="solr.hdfs.blockcache.slab.count">43</int>
         <bool 
name="solr.hdfs.blockcache.direct.memory.allocation">true</bool>
         <int name="solr.hdfs.blockcache.blocksperbank">16384</int>
         <bool name="solr.hdfs.blockcache.read.enabled">true</bool>
         <bool name="solr.hdfs.nrtcachingdirectory.enable">true</bool>
         <int name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">128</int>
         <int name="solr.hdfs.nrtcachingdirectory.maxcachedmb">1024</int>
         <str name="solr.hdfs.home">hdfs://nameservice1:8020/solr8.2.0</str>
         <str name="solr.hdfs.confdir">/etc/hadoop/conf.cloudera.hdfs1</str>
     </directoryFactory>

-Joe

On 8/20/2020 2:30 AM, Prashant Jyoti wrote:
> Hi Joe,
> These are the errors I am running into:
>
> org.apache.solr.common.SolrException: Error CREATEing SolrCore 
> 'newcollsolr2_shard1_replica_n1': Unable to create core 
> [newcollsolr2_shard1_replica_n1] Caused by: Illegal char <:> at index 
> 4: 
> hdfs://hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data/newcollsolr2/core_node3/data\ 
> <http://hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data/newcollsolr2/core_node3/data\>
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1256)
> at 
> org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:93)
> at 
> org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)
> at 
> org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
> at 
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)
> at 
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:420)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:352)
> at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
> at 
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
> at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at 
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
> at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.Server.handle(Server.java:500)
> at 
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
> at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
> at 
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
> at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
> at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
> at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
> at 
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
> at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
> at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.solr.common.SolrException: Unable to create core 
> [newcollsolr2_shard1_replica_n1]
> at 
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1327)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1217)
> ... 47 more
> Caused by: org.apache.solr.common.SolrException: Illegal char <:> at 
> index 4: 
> hdfs://hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data/newcollsolr2/core_node3/data\ 
> <http://hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data/newcollsolr2/core_node3/data\>
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1072)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901)
> at 
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1306)
> ... 48 more
> Caused by: java.nio.file.InvalidPathException: Illegal char <:> at 
> index 4: 
> hdfs://hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data/newcollsolr2/core_node3/data\ 
> <http://hn1-pjhado.tvbhpqtgh3judk1e5ihrx2k21d.tx.internal.cloudapp.net:8020/user/solr-data/newcollsolr2/core_node3/data\>
> at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
> at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
> at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
> at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
> at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
> at sun.nio.fs.AbstractPath.resolve(AbstractPath.java:53)
> at org.apache.solr.core.SolrCore.initUpdateLogDir(SolrCore.java:1380)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:958)
> ... 50 more
>
> Attaching it as a file too. Thanks!
>
> On Wed, Aug 19, 2020 at 9:37 PM Joe Obernberger 
> <[email protected] <mailto:[email protected]>> 
> wrote:
>
>     Your exception didn't come across - can you paste it in?
>
>     -Joe
>
>     On 8/19/2020 10:50 AM, Prashant Jyoti wrote:
>     > You're right Andrew. Even I read about that. But there's a use
>     case for
>     > which we want to configure the said case.
>     >
>     > Are you also aware of what feature we are moving towards instead
>     of HDFS?
>     > Will you be able to help me with the error that I'm running into?
>     >
>     > Thanks in advance!
>     >
>     >
>     > On Wed, 19 Aug, 2020, 5:24 pm Andrew MacKay,
>     <[email protected] <mailto:[email protected]>>
>     > wrote:
>     >
>     >> I believe HDFS support is being deprecated in Solr. Not sure
>     you want to
>     >> continue configuration if support will disappear.
>     >>
>     >> On Wed, Aug 19, 2020 at 7:52 AM Prashant Jyoti
>     <[email protected] <mailto:[email protected]>>
>     >> wrote:
>     >>
>     >>> Hi all,
>     >>> Hope you are healthy and safe.
>     >>>
>     >>> Need some help with HDFS configuration.
>     >>>
>     >>> Could anybody of you share an example of the configuration
>     with which you
>     >>> are running Solr with HDFS in any of your production/dev
>     environments?
>     >>> I am interested in the parts of SolrConfig.xml /
>     Solr.in.cmd/sh which you
>     >>> may have modified. Obviously with the security parts obfuscated.
>     >>>
>     >>> I am stuck at an error and unable to move ahead. Attaching the
>     exception
>     >>> log if anyone is interested to look at the error.
>     >>>
>     >>> Thanks!
>     >>>
>     >>> --
>     >>> Regards,
>     >>> Prashant.
>     >>>
>     >> --
>     >> CONFIDENTIALITY NOTICE: The information contained in this email is
>     >> privileged and confidential and intended only for the use of the
>     >> individual
>     >> or entity to whom it is addressed.   If you receive this
>     message in error,
>     >> please notify the sender immediately at 613-729-1100 and
>     destroy the
>     >> original message and all copies. Thank you.
>     >>
>     >
>
>
>
> -- 
> Regards,
> Prashant.
>
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> 
> 	Virus-free. www.avg.com 
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> 
>
>
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>