[jira] [Resolved] (RAMPART-427) First requests to secured service are failing with "The CLIENT_SIDE parameter is already locked and the value cannot be overridden" error in multi-thread execution.

     [ https://issues.apache.org/jira/browse/RAMPART-427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Lazarski resolved RAMPART-427.
-------------------------------------
    Resolution: Fixed

Fixed in 2.0.0.

  Root cause: RampartMessageData's constructor marked a service as client-side using a
  non-atomic check-then-add of the CLIENT_SIDE parameter. Concurrent first requests to
  a service all observed the parameter as absent, then each added it with
  setLocked(true). The first call succeeded and locked it; the others failed with
  "The CLIENT_SIDE parameter is already locked and the value cannot be overridden",
  causing the first requests to a freshly secured service to fail intermittently under
  concurrent load (matching the report's ~3-4 failures per 100 runs).

  Fix: the parameter is now added by a synchronized, idempotent helper
  (RampartMessageData.addClientSideParameterIfAbsent) that locks on the AxisService and
  re-checks before adding, so it is added at most once regardless of how many threads
  race. The lock is per-service rather than a single global static lock (as in the
  originally attached patch), and the common case - parameter already present - never
  enters the synchronized block, so steady-state message processing stays lock-free.

  Testing: added RampartMessageDataTest with an idempotent case (which alone fails on
  the unfixed code) and a 16-thread concurrent case. The integration test attached to
  this issue was not used - it depended on commons-httpclient 3.x, which is no longer a
  Rampart dependency, and was timing-dependent; the new unit test is deterministic.

  While reviewing the surrounding code, one related robustness fix was also applied:
  non-numeric timestampTTL / timestampMaxSkew values in RampartConfig now raise a clear
  RampartException naming the offending parameter, instead of an unhandled
  NumberFormatException.

  Verified with a full clean 'mvn verify -Papache-release' across all modules,
  including the integration scenarios and the nine policy samples, on
  OpenJDK 17, 21 and 25.
  
  Thanks to Todor Mazgalov for the report and the original patch.

  (Local commits for this issue: 8ce0aeb0 the race fix, c8071bad the review follow-up.
  2.0.0.md still uncommitted on your side.)


> First requests to secured service are failing with "The CLIENT_SIDE parameter is already locked and the value cannot be overridden" error in multi-thread execution.
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: RAMPART-427
>                 URL: https://issues.apache.org/jira/browse/RAMPART-427
>             Project: Rampart
>          Issue Type: Bug
>          Components: rampart-core
>            Reporter: Todor Mazgalov
>            Assignee: Robert Lazarski
>            Priority: Major
>              Labels: Patch
>             Fix For: 2.0.0
>
>         Attachments: rampart-core-add-synchronization-for-CLIENT_SIDE-parameter.patch, rampart-integration-test-testMultiThreadServiceCall.patch
>
>
> First requests to the RampartMessageData in multi-thread execution are failing with error for already locked parameter. This occurs when more than one thread is trying to set CLIENT_SIDE param to the service, the first thread locks the parameter during the addition and the rest cannot override it because it's already locked. I'm providing a patch with test which proves that when threads are more than one a race condition is available during the execution and due to this condition the defect is hard to be reproduced every time. In practise it is failing about 3-4 times for 100 runs (based onto my machine).
> {quote}
> {{monospaced}}
> org.apache.axis2.AxisFault: Error in extracting message properties
>         at org.apache.rampart.handler.RampartSender.invoke(RampartSender.java:76)
>         at org.apache.axis2.engine.Phase.invokeHandler(Phase.java:335)
>         at org.apache.axis2.engine.Phase.invoke(Phase.java:308)
>         at org.apache.axis2.engine.AxisEngine.invoke(AxisEngine.java:250)
>         at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:415)
>         at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:399)
>         at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
>         at org.apache.axis2.client.OperationClient.execute(OperationClient.java:150)
>         at org.apache.axis2.client.ServiceClient.sendRobust(ServiceClient.java:453)
>         at org.apache.axis2.client.ServiceClient.sendRobust(ServiceClient.java:434)
>         at org.apache.rampart.RampartTest$1.run(RampartTest.java:347)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:744)
> Caused by: org.apache.rampart.RampartException: Error in extracting message properties
>         at org.apache.rampart.RampartMessageData.<init>(RampartMessageData.java:381)
>         at org.apache.rampart.MessageBuilder.build(MessageBuilder.java:61)
>         at org.apache.rampart.handler.RampartSender.invoke(RampartSender.java:65)
>         ... 13 more
> Caused by: org.apache.axis2.AxisFault: The CLIENT_SIDE parameter is already locked and the value cannot be overridden.
>         at org.apache.axis2.description.AxisDescription.addParameter(AxisDescription.java:104)
>         at org.apache.rampart.RampartMessageData.<init>(RampartMessageData.java:197)
>         ... 15 more
> org.apache.axis2.AxisFault: Error in extracting message properties
>         at org.apache.rampart.handler.RampartSender.invoke(RampartSender.java:76)
>         at org.apache.axis2.engine.Phase.invokeHandler(Phase.java:335)
>         at org.apache.axis2.engine.Phase.invoke(Phase.java:308)
>         at org.apache.axis2.engine.AxisEngine.invoke(AxisEngine.java:250)
>         at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:415)
>         at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:399)
>         at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
>         at org.apache.axis2.client.OperationClient.execute(OperationClient.java:150)
>         at org.apache.axis2.client.ServiceClient.sendRobust(ServiceClient.java:453)
>         at org.apache.axis2.client.ServiceClient.sendRobust(ServiceClient.java:434)
>         at org.apache.rampart.RampartTest$1.run(RampartTest.java:347)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:744)
> Caused by: org.apache.rampart.RampartException: Error in extracting message properties
>         at org.apache.rampart.RampartMessageData.<init>(RampartMessageData.java:381)
>         at org.apache.rampart.MessageBuilder.build(MessageBuilder.java:61)
>         at org.apache.rampart.handler.RampartSender.invoke(RampartSender.java:65)
>         ... 13 more
> Caused by: org.apache.axis2.AxisFault: The CLIENT_SIDE parameter is already locked and the value cannot be overridden.
>         at org.apache.axis2.description.AxisDescription.addParameter(AxisDescription.java:104)
>         at org.apache.rampart.RampartMessageData.<init>(RampartMessageData.java:197)
>         ... 15 more
> {{monospaced}}
> {quote}
> One possible solution is to be used some synchronization mechanism when the parameter is being locked, I'm attaching a patch which uses "synchronized" block by static object (I suppose that an another synchronization mechanism also can be used). I am not sure whether this parameter really should be set per service and if it is not then a synchronization will not be need because the parameter will not be created for every call but this can be answered in the best way by the Rampart developers.
> rampart-core
> org.apache.rampart.RampartMessageData.java
> {code}
> ...
> if (axisService != null && axisService.getParameter(PARAM_CLIENT_SIDE) != null) {
> 	this.isInitiator = true; //<---- After the addition of the parameter the rest of the threads enter here
> } else {
> 	this.isInitiator = !msgCtx.isServerSide();
> 	// TODO if Axis Service is null at this point, do we have to
> 	// create a dummy one ??
> 	if (this.isInitiator && axisService != null) {
> 		Parameter clientSideParam = new Parameter(); //<---- Two or more threads enter here because the parameter is still not set
> 		clientSideParam.setName(PARAM_CLIENT_SIDE);
> 		clientSideParam.setLocked(true); //<---- Parameter is being locked
> 		msgCtx.getAxisService().addParameter(clientSideParam); //<---- Threads are trying to add the parameter more than once and an exception is generated 
> 	}
> }
> ...
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)