TrueSight Capacity Optimization (TSCO) - How Single lookup vs. Multiple lookup work in TrueSight Capacity Optimization (TSCO) ETLs

Knowledge Article

TrueSight Capacity Optimization (TSCO) - How Single lookup vs. Multiple lookup work in TrueSight Capacity Optimization (TSCO) ETLs
How Single lookup vs. Multiple lookup work in TSCO ETL's
TrueSight Capacity Optimization
Capacity Optimization
TrueSight Capacity Optimization 20* 11* 10*
TrueSight Capacity Optimization
Capacity Optimization
TrueSight Capacity Optimization 20* 11* 10*
In environment are importing data using the vCenter Extractor Service ETL.  Have a separate CMDB that has system configuration information, like to import into TrueSight Capacity Optimization (TSCO).  What is the best way to update the existing entities in TSCO with this additional data?  What about building a new workspace domain hierarchy using existing entities in TSCO based upon relationship data imported through an Object Relationship CSV Parser or Object Relationship SQL Extractor?
 

Introduction

  1. Multiple lookup fields ETLs. These ETLs populate different lookup fields in order to have a more accurate lookup. Some of the out of the box ETLs (like VMware) that populate multiple lookup fields populate also a special lookup field called _COMPATIBILITY_. If you write a custom ETL, it is recommended to populate this field
  2. Single lookup field: the ETL populate a single field, called DEFAULT.

How lookup is performed

  1. Multiple vs. Multiple: the lookup is performed using the field marked as STRONG and WEAK                                                                                          (TSCO 20.02 Understanding entity identification and lookup) (TSCO 20.02 Handling ETL lookup name)
  2. Single vs. Single: the lookup is performed on the field DEFAULT
  3. Multiple vs. Single: the lookup is performed comparing the content of DEFAULT (coming from single lookup ETL) and _COMPATIBILITY_ (coming from multiple lookup ETL)

How populate field in a custom ETL

  1. If you define DS_SYSNAME giving only a value (e.g. host name) the ETL will create single lookup field DEFAULT populated with that value
  2. If you define DS_SYSNAME with a sequence like this LOOKUPFIELD1#LOOKUPVALUE1##LOOKUPFIELD2#LOOKUPVALUE2 and after define in STRONGLOOKUPFIELDS and WEAKLOOKUPFIELDS the kind of field, you will create a multiple lookup
Just to give you an example, this is how the VMware ETL populate these values
DS_SYSNM=HOSTNAME#vl-pun-bcm-dv20##PARENT_VCNAME#Vcenter name##NAME#VMname##PARENT_HOSTNAME#hostname.bmc.com##_COMPATIBILITY_#564da903-e29e-976a-6970-6164020b0d3b##UUID#564da903-e29e-976a-6970-6164020b0d3b##VMW_VMREF#vm-1234##PARENT_VCUUID#63BA5246-A098-407F-B797-89E5E1B145D4;
STRONGLOOKUPFIELDS=PARENT_VCUUID&&VMW_VMREF##PARENT_VCNAME&&VMW_VMREF
WEAKLOOKUPFIELDS=UUID##HOSTNAME##PARENT_HOSTNAME&&NAME##PARENT_VCNAME&&PARENT_CLUSTERNAME&&NAME##NAME


A common use case: VMware ETL and custom CMDB

A common use case is the integration of a custom CMDB that share entity catalog with VMware ETL

There are two ways to do it:

  1. Using multiple lookup
    If you are writing a custom ETL or your are using Object Relationship SQL Extractor, you can use the multiple lookup feature.
  2. Using single lookup field
    If you are using Object Relationship CSV that doesn't support multiple lookup field or there are conditions in which you have to share the lookup with another ETLs that doesn't support multiple lookup, you have to change the configuration of VMware ETL in order to populate the field _COMPATIBILITY_ with system name or host name, as showed in this screenshot
  3.  
  4. 1

 

When looking up an entity the basic rules are:
  (A)  If an entity only reports the DEFAULT lookup (this is called 'single lookup') then when looking for a matching entity in the shared catalog BHCO will look for a matching DEFAULT lookup or a matching _COMPATIBILITY_ lookup within the matching entity.
  (B) If an entity reports a differently named lookup field or multiple lookup fields (both of these are called 'multiple lookup') then when looking for a matching entity in the shared catalog BHCO will look for a matching set of lookups based upon the specified STRONGLOOKUPFIELDS for the ETL.

 

Q: Can the DEFAULT lookup be defined as a 'multiple' lookup field?

There is a convention that only 'single lookup' ETLs use the DEFAULT lookup field.  So, it would be breaking a norm for a multiple lookup ETL to define a lookup field called DEFAULT.  Beyond that, it generally wouldn't be considered a best practice to define DEFAULT as a STRONGLOOKUPFIELD in a multiple lookup ETL, although the product wouldn't prevent it from being done.

 

Q: Which is better a better choice when designing an ETL: single or multiple lookup?

 

In general single lookup is a less safe way to define the lookup fields for entities because it leads most easily to entity collisions (that is where two real world separate entities end up sharing the same sysid).  When you have ETLs importing just the DEFAULT lookup you can get into a situation where an insufficiently unique lookup value causes multiple entities to end up sharing the same sysid and there is no good way to split entities.  The only option after an entity collision is to fix the underlying lookup problem that caused the entity collision, delete the entity, and then let the ETLs create new entities on their next run.

 

Q: In an environment with both single lookup and multiple lookup ETLs what is the best way to allow the single lookup ETLs to map to a multiple lookup ETL created entity?

 

If you have entities reporting data using 'single lookup' (the DEFAULT lookup field) and other entities reporting data using 'multiple lookup' (one or more named named lookup fields) then for the multiple lookup ETLs you usually would want to import an _COMPATIBILITY_ lookup if you wanted the single lookup ETLs to be able to map data to the multiple lookup ETLs.
TrueSight Capacity Optimization
Capacity Optimization
TrueSight Capacity Optimization 20* 11* 10*
Attachment(s):