Knowledge Article
In TrueSight Capacity Optimization (TSCO) how to use the Data Provider API to extract data for import into a 3rd party tool the same way as the Capacity Views in TrueSight Presentation Server (TSPS)?
In TSCO how to use the Data Provider API to extract data for import into a 3rd party tool the same way as the Capacity Views in TSPS?
TrueSight Capacity Optimization
Capacity Optimization
TrueSight Capacity Optimization 20.02, 11.5, 11.3.01, 11.0, 10.7
The TSCO Datahub supports a "Data Provider" service which can be used to query data from TSCO. The "Data Provider" API is described in the product documentation. But, the supported API described in the product documentation is only able to access data for Materialized Views but a lot of the data provided in the TSCO Capacity Views in TSPS comes from "template Datamarts" which is only accessible via a Private API. NOTE: The curl examples below are showing the curl command format on Linux. Running the curl command on Windows will require different quoting and double-quote escaping. |
QuestionThe TSCO Datahub supports a "Data Provider" service which can be used to query data from TSCO. The "Data Provider" API is described in the product documentation. But, the supported API described in the product documentation is only able to access data for Materialized Views but a lot of the data provided in the TSCO Capacity Views in TSPS comes from "template Datamarts" which is only accessible via a Private API.Answer In TSCO the Capacity Views use a private API to access Datamart data. It is possible to use the supported Data Provider API to extract the datamart result.
An example curl command to extract data from Datmart 2030 is: curl -H 'Authorization: Bearer eyJ0eXAiOiJK...cut_long_token...7d0UKdzA' -H 'Content-Type: application/json' -X POST -d '{"filters":[{"name":"timefilterlastxdays","condition":"EQUALS","value":"D30"}],"options":{"pagenum":0,"pagesize":10}}' -k https://<TSPS-Console>:8280/dh-services/data-prv/v2/datamarts/2030/data -o 2030.json If you don't mention the parameter -d and filters, then the default pagination will apply as below: "pagination" : { "has.more.data" : true, "pageNumber" : 0, "pageSize" : 50 }, Q: How can I see the data fields (columns) of the datamart? The following curl command will display the data field definitions and then the extracted data from the API. curl -H 'Authorization: Bearer eyJ0eXAiOiJK...cut_long_token...7d0UKdzA' -H 'Content-Type: application/json' -X POST -d '{"filters":[{"name":"timefilterlastxdays","condition":"EQUALS","value":"D30"}],"aggregations":[{"name":"*","operation":"count","displayAs":"table_rows_count"}]}' -k https://<TSPS-Console>:8280/dh-services/data-prv/v2/datamarts/2030/aggregatedata -o 2030_agg_data.json At the top, you will get the metadata: { "data.format" : "EXTENDED", "href" : "https://hostname:8280/dh-services/data-prv/v2/datamarts/2030/aggregatedata", "metadata" : { "percentage.upperbound" : 1 }, Then, you will see a list of all the columns (data fields) of the datamart: "datafields" : [ { "name" : "avgvalue", "label" : "Avgvalue", "meaning" : "SYSID", "primary" : false, "datatype" : "NUMBER" }, { "name" : "duration", "label" : "Duration", "primary" : false, "datatype" : "NUMBER" }, { "name" : "maxvalue", "label" : "Maxvalue", "primary" : false, "datatype" : "NUMBER" }, { "name" : "minvalue", "label" : "Minvalue", "primary" : false, "datatype" : "NUMBER" }, { "name" : "sumvalue", "label" : "Sumvalue", "primary" : false, "datatype" : "NUMBER" }, { "name" : "sysmetricid", "label" : "Sysmetricid", "primary" : false, "datatype" : "NUMBER" }, { "name" : "ts", "label" : "Ts", "meaning" : "TS", "primary" : false, "datatype" : "DATE" } ], and then the list of filters applied and it's metadata followed by total number of rows found, datamart name and aggregation method: "standardfiltermetadata" : { "timefilter" : { "name" : "timefilter", "label" : "Time filter label", "description" : "Time filter description", "filtertype" : "time-filter" }, "tagfilter" : { "name" : "tagfilter", "label" : "Tag filter label", "description" : "Tag filter description", "filtertype" : "tags-filter" }, "domainfilter" : { "name" : "domainfilter", "label" : "Domain filter label", "description" : "Domain filter description", "filtertype" : "domain-filter" } }, "filters" : [ { "name" : "@{lastxdays}", "value" : "D30", "note" : "This filter will be applied on column ts" } ], "data" : [ { "table_rows_count" : "154318" } ], "name" : "citidm1", "aggregations" : [ { "name" : "*", "operation" : "count", "displayAs" : "table_rows_count" } ] Q: Can another method, such as a Java program, be used to query the API instead of 'curl'? TSCO Technical Support wouldn't be able to provide additional information on how you would do the same process using Java -- but it is basically the same methodology since it would be querying an HTTP-based API to extract JSON formatted data. So, something certainly could be written in Java to connect to the same API URL and then parse out the JSON object that is returned. But Technical Support wouldn't be able to provide any example code of how to do that. One might be able to get something from someone on the BMC Communities forum (https://communities.bmc.com/community/bmcdn/capacity_management) but that type of request might be even beyond the scope of the BMC Communities since usually, that forum would usually get to the level of describing the API but not go into details on the chosen implementation of how to extract data from the API.
There are a bunch of examples online regarding creating a Java REST API client but it would require someone familiar with java programming and the 3rd party packages they use (Apache HttpClient, Jersey project) to really say whether those examples would be useful for any given data extraction project. Q: What information is available in the output from the Private API in relation to what data is being reported? At the top of the output you will get metadata of the datamart that is being extracted:
{ "data.format" : "EXTENDED", "href" : "https://clm-aus-ty0tbs.bmc.com:8280/dh-services/data-prv/v2/datamarts/2030/data", "metadata" : { "percentage.upperbound" : 1 }, "pagination" : { "has.more.data" : true, "pageNumber" : 0, "pageSize" : 10 }, "filters" : [ { "name" : "@{lastxdays}", "value" : "D30", "note" : "This filter will be applied on column ts" } ], Then you will get a record that looks like this: "data" : [ { "sysmetricid" : "20316", "ts" : "2020-10-21 00:00:00", "duration" : "86400", "avgvalue" : "0.99777", "minvalue" : "0.9905", "maxvalue" : "0.99945", "sumvalue" : "23.94663" }, { "sysmetricid" : "20316", "ts" : "2020-10-22 00:00:00", "duration" : "86400", "avgvalue" : "0.99761", "minvalue" : "0.99041", "maxvalue" : "0.99909", "sumvalue" : "23.94265" }, { "sysmetricid" : "20316", "ts" : "2020-10-23 00:00:00", "duration" : "86400", "avgvalue" : "0.99773", "minvalue" : "0.99011", "maxvalue" : "0.99965", "sumvalue" : "23.9457" }, { "sysmetricid" : "20316", "ts" : "2020-10-24 00:00:00", "duration" : "86400", "avgvalue" : "0.99802", "minvalue" : "0.99326", "maxvalue" : "0.99902", "sumvalue" : "23.95258" }, { "sysmetricid" : "20316", "ts" : "2020-10-25 00:00:00", "duration" : "86400", "avgvalue" : "0.99787", "minvalue" : "0.99119", "maxvalue" : "0.9995", "sumvalue" : "23.94888" }, { "sysmetricid" : "20316", "ts" : "2020-10-26 00:00:00", "duration" : "86400", "avgvalue" : "0.99778", "minvalue" : "0.992", "maxvalue" : "0.99917", "sumvalue" : "23.94679" }, { "sysmetricid" : "20316", "ts" : "2020-10-27 00:00:00", "duration" : "86400", "avgvalue" : "0.99775", "minvalue" : "0.99215", "maxvalue" : "0.99916", "sumvalue" : "23.9461" }, { "sysmetricid" : "20316", "ts" : "2020-10-28 00:00:00", "duration" : "86400", "avgvalue" : "0.99793", "minvalue" : "0.99206", "maxvalue" : "0.99935", "sumvalue" : "23.95035" }, { "sysmetricid" : "20316", "ts" : "2020-10-29 00:00:00", "duration" : "86400", "avgvalue" : "0.998", "minvalue" : "0.99268", "maxvalue" : "0.99894", "sumvalue" : "23.95206" }, { "sysmetricid" : "20316", "ts" : "2020-10-30 00:00:00", "duration" : "86400", "avgvalue" : "0.99801", "minvalue" : "0.99389", "maxvalue" : "0.99921", "sumvalue" : "23.95241" } ] So when you are using the private API a list of the metadata (and the filters applied) is reported and then a list of the actual data (which corresponds to a row of data in the table). Q: Is it possible to test access to the Private API via a Web Browser? A browser isn't going to be able to access that URL because it requires a POST call and a browser is going to make a GET call.
So you could access that URL with a browser plugin like Postman but you won't be able to access it directly by calling the URL. That is probably the reason that you aren't able to access the URL via Java -- you are probably making a Java call that does an HTTP GET rather than an HTTP POST. Q: How do I control the volume of data being returned by the Private API call? You need to specify the "pagesize" option to select the number of rows returned. If you have not mentioned any value for it, then you will see 50 rows returned (50 is the default value for pagesize)
For example: curl -H 'Authorization: Bearer <Auth token>' -H 'Content-Type: application/json' -X POST -d '{"filters":[{"name":"timefilterlastxdays","condition":"EQUALS","value":"D30"}],"options":{"pagenum":0,"pagesize":10}}' -k https://<TSPS-Console>:8280/dh-services/data-prv/v2/datamarts/2030/data -o 2030.json Note: The '-d' parameter that gives the filters and options ('pagenum' and the 'pagesize' values). The original output provided by Development is this: -d '{"filters":[{"name":"@{lastxdays}", “value":"D30"}],"options":{"pagenum":0,"pagesize":20}}' For example, that 'filters' section is what is necessary to select the time period for the data (Last 30 Days, Last 7 Days, or Last 1 Day). If you don't specify that 'filters' parameter in your API call then it defaults to the "Last 30 Days" data. If you don't specify the pagesize option then it defaults to the first 50 entities. Q: Is API_DATA_PROVIDER read-only or read-write activity? API_DATA_PROVIDER is a read+write activity.
However, the datamarts API can be used also with WEB_DASHBOARD/API_DATA_READ which are read-only activities. Q: Anything else to consider when using curl to access the APIs? (a) If the '-d' flag hasn't been specified to define the extaction parameters it is necessary to provide the '-X POST' flag or the curl command will just silently fail to extract any data. Running the curl command without the '-d [data]' parameter and without the '-X POST' will show the API call failing in the $CPITBASE/apache2/logs/access_log.YYYY-MM-DD.log with a message like this: 192.0.2.100 - - [21/Nov/2023:09:50:56 -0600] "GET /dh-services/data-prv/v2/datamarts/1624/data HTTP/1.1" 405 - "-" "curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.44 zlib/1.2.3 libidn/1.18 libssh2/1.4.2" 3 That '405' error is a "Method not allowed" error which occurs due to the API call receiving a GET request when it requires a POST request. (b) If there are problems with the bearer token being used an error 401 or 403 should be returned Using a token that isn't valid will result in a response like this: <!doctype html><html lang="en"><head><title>HTTP Status 401 – Unauthorized</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 401 – Unauthorized</h1></body></html> Using a token generated against a Role that doesn't have access to the data will result in a response like this: <!doctype html><html lang="en"><head><title>HTTP Status 403 – Forbidden</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 403 – Forbidden</h1></body></html> (b) If curl returns no response it can be useful to run the curl command with the '--verbose' flag. That will output header information which will include information regarding the HTTP response code. The HTTP response code will be useful for trying to understand what problem curl is having accessing the API data. |
Attachment(s):