Knowledge Article

Article Number

000260566

Old Article Number

000085299

Article Type

Solutions to a Product Problem

Title

TrueSight Server Automation (TSSA) : Error while executing NSH or NSH script job: nsh 9764 child_info_fork::abort: cygblcrypto-0.9.8.dll: Loaded to different address: parent(0x720000) != child(0x8D0000)

Summary

Error while executing NSH or NSH script job: nsh 9764 child_info_fork::abort: cygblcrypto-0.9.8.dll: Loaded to different address: parent(0x720000) != child(0x8D0000)

Product

TrueSight Server Automation Suite

Component

TrueSight Server Agent and NSH

Applies to

TrueSight Server Automation Suite

Problem

One or more of the following errors occur executing the NSH or NSH Script Job

nsh 9764 child_info_fork::abort: cygblcrypto-0.9.8.dll: Loaded to different address: parent(0x720000) != child(0x8D0000)

---------------------------------------------------

nsh 4556 child_info_fork::abort: D:\Program Files\BMC Software\BladeLogic\NSH\bin\cygblcrypto-0.9.8.dll: Loaded to different address: parent(0x6B0000) != child(0x6D0000)

---------------------------------------------------

nsh 4964 child_info_fork::abort: C:\Program Files\BMC Software\BladeLogic\NSH\bin\cygblcrypto-0.9.8.dll: Loaded to different address: parent(0x6D0000) != child(0x8D0000)
nsh: fork failed: resource temporarily unavailable

---------------------------------------------------

Error   05/19/2019 17:24:14   1 [main] nsh (5588) C:\Program Files\BMC Software\BladeLogic\NSH\bin\nsh.exe: *** fatal error - cygheap base mismatch detected - 0x1157408/0x10B7408.
Error   05/19/2019 17:24:14   This problem is probably due to using incompatible versions of the cygwin DLL.
Error   05/19/2019 17:24:14   Search for cygwin1.dll using the Windows Start->Find/Search facility
Error   05/19/2019 17:24:14   and delete all but the most recent version. The most recent version *should*
Error   05/19/2019 17:24:14   reside in x:\cygwin\bin, where 'x' is the drive on which you have
Error   05/19/2019 17:24:14   installed the cygwin distribution. Rebooting is also suggested if you
Error   05/19/2019 17:24:14   are unable to find another cygwin DLL.
Error   05/19/2019 17:24:14   nsh:13: fork failed: resource temporarily unavailable
Error   05/19/2019 17:24:14   1 [main] nsh 10936 fork: child -1 - forked process 5588 died unexpectedly, retry 0, exit code 0xC0000142, errno 11

Cause

Cygwin, BLODA software interfering with operation

Solution

NSH on Windows relies on Cygwin. For more information on the Cygwin fork failures refer Faq.using.fixing-fork-failures

There are a few options for dealing with these errors.

1. TSSA 21.3 comes with 64bit NSH (thus 64bit Cygwin). Because of the increased 64bit address space this error should be less likely to happen in TSSA 21.3 and later. UpgradeTSSA to 21.3+.
2. Where upgrade is not possible, or the problem still occurs in 21.3+, first option is to use the Cygwin rebase utility to alter the base memory address used by the Cygwin libraries so they will not conflict.
3. Another approach is to white list the Cygwin libraries and executables in any AntiVirus or Host Intrusion Detection software on the system (BLODA software: https://cygwin.com/faq/faq.html#faq.using.bloda)

Either procedure is performed on the system with NSH installed and the error occurs. This could be the TSSA application server, systems with the RCP client installed, or TSO servers integrated with TSSA.

Rebase:

Please run the below steps wherever the NSH that encounters this error is installed (AppServer, RCP/NSH client).

The rebase utility is in the NSH\bin directory

On the system throwing the fork errors:

Stop all TSSA applications/services: AppServer, ProcessSpawner, PXE, TFTP, BSA Console, NSH, RSCD. This includes any NSH-related binaries such as sed, awk, etc
- An easy way to ensure nothing related to TSSA is running is to set any TSSA related startup services to manual and reboot the system.
Open a cmd shell. If not logged in as the local built-in Administrator account, use the Run as Administrator option when opening the cmd shell.
In the Administrator cmd shell, change directory to the NSH\bin folder.
Run the command (For 64 bit Cygwin in 21.3 or later, use the address of 0x200000000 or larger):
rebase.exe -b 0x70000000 bl*.dll cygblcrypto*.dll cyggcc*.dll cygblssl*.dll

Output like below is seen:

bladmin.dll: skipped because wrong machine type.
blclientjni.dll: skipped because not rebaseable
blclientjni64.dll: skipped because wrong machine type.
blclijni.dll: skipped because not rebaseable
blinexclude.dll: skipped because wrong machine type.
bllibeay64.dll: skipped because not rebaseable
blsrp.dll: skipped because not rebaseable
blssleay32.dll: skipped because not rebaseable
blssleay64.dll: skipped because wrong machine type.

Start the TSSA Services on the system and retest.

If the same fork errors continues, then try the following steps:

1) Run the rebase with i option to find current memory range:

Example)

C:\Program Files\BMC Software\BladeLogic\NSH\bin>rebase -i bl*.dll cygblcrypto*.dll cyggcc*.dll cygblssl*.dll
bladmin.dll: skipped because wrong machine type.
blclientjni64.dll: skipped because wrong machine type.
blclijni.dll: skipped because wrong machine type.
blinexclude.dll: skipped because wrong machine type.
bllibeay64.dll: skipped because wrong machine type.
blsrp.dll: skipped because wrong machine type.
blssleay64.dll: skipped because wrong machine type.
/usr/bin/blclientjni.dll                      base 0x10000000 size 0x00008000
/usr/bin/blcygsrp.dll                         base 0xa0000000 size 0x000a2000
/usr/bin/bllog4c.dll                          base 0xa00b0000 size 0x000af000
/usr/bin/blncdll.dll                          base 0xa0160000 size 0x0018b000
/usr/bin/blncursesdll.dll                     base 0xa02f0000 size 0x000ce000
/usr/bin/cygblcrypto-1.0.0.dll                base 0xa03c0000 size 0x00289000
/usr/bin/cyggcc_s-1.dll                       base 0xa0650000 size 0x00021000
/usr/bin/cygblssl-1.0.0.dll                   base 0xa0680000 size 0x0007f000

2) Repeat the above steps with the following memory ranges. Below is an example of rebase starting at 0x50000000

Example)

C:\Program Files\BMC Software\BladeLogic\NSH\bin>rebase -b 0x50000000 bl*.dll cygblcrypto*.dll cyggcc*.dll cygblssl*.dll
bladmin.dll: skipped because wrong machine type.
blclientjni.dll: skipped because not rebaseable
blclientjni64.dll: skipped because wrong machine type.
blclijni.dll: skipped because wrong machine type.
blinexclude.dll: skipped because not rebaseable
bllibeay64.dll: skipped because wrong machine type.
blsrp.dll: skipped because not rebaseable
blssleay64.dll: skipped because not rebaseable

C:\Program Files\BMC Software\BladeLogic\NSH\bin>rebase -i bl*.dll cygblcrypto*.dll cyggcc*.dll cygblssl*.dll
bladmin.dll: skipped because wrong machine type.
blclientjni64.dll: skipped because wrong machine type.
blclijni.dll: skipped because wrong machine type.
blinexclude.dll: skipped because wrong machine type.
bllibeay64.dll: skipped because wrong machine type.
blsrp.dll: skipped because wrong machine type.
blssleay64.dll: skipped because wrong machine type.
/usr/bin/blclientjni.dll                      base 0x10000000 size 0x00008000
/usr/bin/blcygsrp.dll                         base 0x50000000 size 0x000a2000
/usr/bin/bllog4c.dll                          base 0x500b0000 size 0x000af000
/usr/bin/blncdll.dll                          base 0x50160000 size 0x0018b000
/usr/bin/blncursesdll.dll                     base 0x502f0000 size 0x000ce000
/usr/bin/cygblcrypto-1.0.0.dll                base 0x503c0000 size 0x00289000
/usr/bin/cyggcc_s-1.dll                       base 0x50650000 size 0x00021000
/usr/bin/cygblssl-1.0.0.dll                   base 0x50680000 size 0x0007f000

3) Increment the memory range, if the issue persists:

Example)

rebase.exe -b 0x50000000 bl*.dll cygblcrypto*.dll cyggcc*.dll cygblssl*.dll

rebase.exe -b 0x60000000 bl*.dll cygblcrypto*.dll cyggcc*.dll cygblssl*.dll

rebase.exe -b 0x70000000 bl*.dll cygblcrypto*.dll cyggcc*.dll cygblssl*.dll

rebase.exe -b 0x80000000 bl*.dll cygblcrypto*.dll cyggcc*.dll cygblssl*.dll

rebase.exe -b 0x90000000 bl*.dll cygblcrypto*.dll cyggcc*.dll cygblssl*.dll

rebase.exe -b 0xA0000000 bl*.dll cygblcrypto*.dll cyggcc*.dll cygblssl*.dll

If below error is seen after following the above steps then please make sure that the appserver is stopped and no other TSSA processes are running while executing rebase.exe

User-added image

Text:
bladmin.dll: skipped because wrong machine type.
...
...
blssleay64.dll: skipped because not rebasable
ReBaseImage (/usr/bin/blcygsrp.dll) failed with last error = 6

The last error in the screenshot above may occur when multiple versions of the cygwin DLL are loaded by different programs on the system. Search for cygwin1.dll using the Windows Start->Find/Search facility and replace older versions with the latest present on the system.

The rebase procedure may need to be run after apply os patches and/or reboots and is not a permanent fix for the problem.

Whitelisting

It has been observed that some security software as Antivirus, Anti-Malware, Anti-Spyware, etc. can interfere with Cygwin functionality and cause random unexpected behavior as noted in Cygwin documentation:faq.using.bloda . Typically these software provide some method to whitelist files and exempt them from inspection. The attached TSSA_Whitelist.txt contains a list of files to whitelist in said applications.

Alternatively, the security application may have configuration settings to change its behavior. For example, CrowdStrike's Falcon software has a setting called Additional User Mode Data. Having this mode enabled may cause the noted problem with Cygwin. Disabling it should prevent the Cygwin errors. Other software may have similar configuration settings that will avoid the conflict with Cygwin.

Attachment(s):

TSSA_WhiteList.txt
rebase.zip