BREPipelineFrameworkComponent - An item with the same key has already been added

Sep 16, 2014 at 3:10 AM
I am attempting to reproduce an error I am getting in the BREPipelineFrameworkComponent but wanted to check something first.

In class BREPipelineMetaInstructionCollection the metaInstructionCollection private field is declared as a SortedDictionary. I understand that an instance member of type SortedDictionary is not thread safe and could possibly be causing my problem (to be verified).

My question is why use a non-thread safe collection object? I don't know for sure yet if this is my problem but it does have the potential to cause the "An item with the same key has already been added" exception.

Cheers,
Simon
Coordinator
Sep 16, 2014 at 4:47 AM
Hi Simon,

The choice to use the SortedDictionary class is a legacy decision as that used to be the only means of ordering MetaInstructions, and still is the means in v1.5+ of the framework if you set the pipeline component parameter InstructionExecutionOrder to Legacy. I would eventually like to get rid of this altogether since I don't think it will have much of an impact if we move away from the legacy ordering altogether.

That said, thread safety really shouldn't be an issue since the collection will never be shared across threads. There are also checks when adding to this collection to see if items with the same key already exist or not, so I am doubtful whether this is the issue.

Could you use the CAT Instrumentation Framework Controller to enable tracing for pipeline components and reproduce this issue to see if the tracing logs give us any more hints as to the cause/location of the problem? Otherwise if you can share more details (i.e. specific rules and messages you are using to reproduce this issue) then I can debug it further for you.

Cheers
Johann
Sep 16, 2014 at 5:25 AM
Thanks for the prompt reply, Johann.

I have used CAT to determine the last trace message which is "Reading stream to ensure it's read logic get's executed prior to pipeline component execution". I am still trying to reproduce the issue on my development machine. The next trace statement is usually "Adding MetaInstruction BREPipelineFramework.SampleInstructions.MetaInstructions.CachingMetaInstructions to Execution Policy facts" but this is not reached.

I hope to reproduce the issue soon. If I can then I'll be able to find exactly where the issue is.

Cheers,
Simon
Sep 16, 2014 at 5:48 AM
Another thought - I have upgraded to the latest BRE pipeline framework and used it for a new receive pipeline. I have an older receive pipeline where I haven't updated the binding. In the BizTalk Administration Console pipeline properties dialog the old one displays only the 5 properties rather than the 11 of the latest version. This could cause the issue. Best I update the old pipeline binding first.
Coordinator
Sep 16, 2014 at 10:16 AM
Thanks Simon, yes it's a good idea to update the binding to see if the problem still occurs. Let me know.

Cheers
Johann
Sep 16, 2014 at 11:07 PM
I have determined that the error occurs in method CopyMessageParts in class BREPipelineFrameworkComponent where a part is added:
 partNames.Add(partCounter, partName);
        private void CopyMessageParts(IBaseMessage sourceMessage, IBaseMessage destinationMessage, IBaseMessagePart newBodyPart)
        {
            string bodyPartName = sourceMessage.BodyPartName;
            for (int partCounter = 0; partCounter < sourceMessage.PartCount; ++partCounter)
            {
                string partName = null;
                IBaseMessagePart messagePart = sourceMessage.GetPartByIndex(partCounter, out partName);
                partNames.Add(partCounter, partName);

                if (partName != bodyPartName)
                {
                    destinationMessage.AddPart(partName, messagePart, false);
                }
                else
                {
                    destinationMessage.AddPart(bodyPartName, newBodyPart, true);
                }
            }
        }
The part being added has a key of "0" and a name of "body".

My scenario is
  • Receive pipeline with two stages
  • Stage 1: Xml disassembler
  • Stage 2: BREPipelineFrameworkComponent with default properties (apart from an ApplicationContext)
  • Receive an enveloped message with > 1 item
  • Message is debatched in the disassembler
  • Individual items are processed through the BRE pipeline component
I don't think the rule policies come in to play at this stage.

Corrected code - wrapped .Add with a Contains check:
        private void CopyMessageParts(IBaseMessage sourceMessage, IBaseMessage destinationMessage, IBaseMessagePart newBodyPart)
        {
            string bodyPartName = sourceMessage.BodyPartName;
            for (int partCounter = 0; partCounter < sourceMessage.PartCount; ++partCounter)
            {
                string partName = null;
                IBaseMessagePart messagePart = sourceMessage.GetPartByIndex(partCounter, out partName);

                if (!partNames.ContainsKey(partCounter))
                {
                    partNames.Add(partCounter, partName);
                }

                if (partName != bodyPartName)
                {
                    destinationMessage.AddPart(partName, messagePart, false);
                }
                else
                {
                    destinationMessage.AddPart(bodyPartName, newBodyPart, true);
                }
            }
        }
My concern is any flow-on effects of this change.

Cheers,
Simon
Coordinator
Sep 22, 2014 at 10:37 AM
Thanks Simon, funnily enough I encountered the same problem today. Will evaluate your fix and patch shortly.

Cheers
Johann
Coordinator
Sep 22, 2014 at 11:54 AM
Hi Simon,

I've fixed the problem by clearing the partNames dictionary at the beginning of the CopyMessageParts method. I think that's probably cleaner than your suggestion (no offence) since it means we can be assured that the collection will only contain message part names pertaining to the current message (it appears that the collection persists from the first run of the pipeline component processing such that when it starts processing the second debatched message the collection is still populated with the first message's values). I've re-run all the unit tests and all looks good from a regression perspective.

I will look at packaging this in the next few days along with the other bug fixes that you have reported to me recently. Thanks for the help.

Cheers
Johann
Coordinator
Sep 23, 2014 at 1:42 AM
Hi Simon,

Please try v1.5.2 of the framework and let me know if that works for you. You will just need to uninstall the old version from the add/remove programs MMC, and then install the new version. No changes required to vocabularies etc... You'll of course have to restart host instances/IIS app pools.

The issue you raised about messages with no namespaces throwing an exception in the MessageUtility class has also been addressed.

Thanks for your help pinpointing and addressing these bugs.

Cheers
Johann
Marked as answer by jcooper1982 on 4/20/2015 at 9:06 PM
Apr 26, 2016 at 12:09 PM
Edited Apr 26, 2016 at 12:11 PM
EDIT: Forgot to mention that I use version 1.6 of the framework

Hello,

Sad to inform that I occasionally get this error when "HelperMetaInstructions.TransformMessage" is executed. I got it a few times in testing and did not think much of it, but today this error popped up in production as well, which of course raised some heads. Very grateful for any input on this issue. See error message below.

Greetings,

Martin Gjerde

Error Information:
There was a failure executing the receive pipeline: "ZZZ.BizTalk.Diagnostics.Pipelines.ReceivePAFInterchange, ZZZ.BizTalk.Diagnostics.Pipelines, Version=1.0.0.0, Culture=neutral, PublicKeyToken=e421ce17f60f1547" Source: "BREPipelineFrameworkComponent" Receive Port: "ZZZ.BizTalk.Diagnostics.PAF.Rcv" URI: "\vs040\KFY_Remiss_Svar*.xml" Reason: Exception encountered while executing BRE policy ZZZ.BizTalk.Diagnostics.Transformation. Top level exception message - An exception occurred when invoking class member BREPipelineFramework.SampleInstructions.MetaInstructions.HelperMetaInstructions.TransformMessage in assembly BREPipelineFramework.SampleInstructions, Version=1.0.0.0, Culture=neutral, PublicKeyToken=83eab0b166470ebc.
Innermost exception was - System.ArgumentException: An item with the same key has already been added.
at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
at BREPipelineFramework.Helpers.ObjectCreator.ResolveType(String fullyQualifiedClass)
at BREPipelineFramework.SampleInstructions.MetaInstructions.HelperMetaInstructions.TransformMessage(String mapClassName, String mapAssemblyName, TransformationSourceSchemaValidation validation)

Wed at 2:54 PM
Experiencing exactly the same exception (once in a while) as McNutty.
Not very acceptable in production...