vSphere数据采集器

名称:vSphere数据采集器

描述:iTop CMDB中的采集器与Vsphere对接数据同步(VM,虚拟化主机,群集)

版本:1.0.13

发布:2020-07-07

码:适用于vSphere的combodo-数据-采集器

州:稳定

用名称:vSphere数据采集器

github 模块 1:https:github.commCombodooitop-数据-采集器-base

github 模块 2:https:github.commCombodooitop-数据-采集器-vsphere

扩散:iTop集线器

此独立的应用从vSphere服务器(使用vSphere Web Services)收集有关整个数据中心的信息,并使用多个同步数据源将这些信息与iTop同步。

vSphere Data Collector connections

特征

  • 服务器(及其品牌,模型),虚拟机管理程序(及其OS系列和OS版本),服务器场,虚拟机(及其OSFamily和OS版本)的自动化库存。
  • IPv4地址和逻辑接口的可选库存
  • 采集器可以驻留在任何具有通过Web访问vSphere Web Services和iTop的系统上。
  • 在iTop中自动创建和更新同步数据源。
  • 从版本1.0.12开始,收集机制可以以某种方式扩展
  • 该采集器利用iTop的内置数据同步机制。有关数据同步如何工作的更多信息,请参考数据同步概述并依靠基础数据采集器机制

修订记录

发布日期版本注释 
2020-07-071.0.13-区别未安装teemip和itop REST API问题的错误日志
-多配置文件
-新的CSV采集器
-在日志中添加了可配置的时间戳
-用法的新选项:–帮助
 
2020-04-021.0.12-使服务器和虚拟化主机收集器可配置
-修复了由空白数据存储名称引起的崩溃(感谢ITOMIG的DavidWißen报告了该问题)。
-修复了一些OpenVM Tools报告IP地址带有尾随空格的问题(感谢ITOMIG的Martin Raenker)。
 
2018-12-311.0.11纠正1.0.10引入的回归:
-改进了对iTop 2.4+(废弃标志)的支持
-删除了调试跟踪
-修复跳过“ connectionState”未“连接”的虚拟机
 
  2018-08-241.0.10处理TeemIp:
-自动检测是否存在TeemIp(作为iTop模块或作为独立的应用)。
-IPv4地址的可选同步
-逻辑接口的可选同步
 
2018-07-021.0.9坚固性:
-在调试模式下有更多跟踪(–console_log_level = 9)
-跳过“ connectionState”未“连接”的虚拟机,因为它们的大多数属性不可访问,并且会导致PHP/Soap崩溃(带有核心转储!)
 
2018-02-221.0.8-os_版本_mapping仅用于虚拟机监控程序,使其也适用于VirtualMachines。
在正则表达式中正确支持UTF-8字符(代码现在始终使用u修饰符)
 
2017-04-241.0.7修复了VM的状况的价值:“生产”而不是“活跃”。 
2017-03-081.0.6-代码清理。
-数据同步,状况和full_load_interval现在是所有数据源的同质且可配置的参数。
-OS版本和OS系列都有自己的映射表。
-修复了虚拟化主机状况价值(现在默认为“生产”)的问题。
-添加了check_soap.php脚本进行故障排除。
-防止ESX断开连接时PHP崩溃。
-不要收集IPV6作为虚拟机的管理IP地址(因为该字段只能包含IPV4)。非常感谢Andrew Armstrong提出此修复程序。
 
2016-12-081.0.5绕过SSL证书验证和自动检测无效证书的新选项。 
2015-11-161.0.4用虚拟机描述内的空格替换换行符。将vCPU数量固定为+1 
2015-04-171.0.3添加了虚拟机“描述”字段的集合 
2015-02-231.0.2必须在管理程序之前加载服务器 
2015-01-071.0.1Beta版本 
2014-05-131.0.0首款Alpha版本 

局限性

数据-采集器-for-VSphere在vCenter 7.0.0上不起作用

采集器的版本不收集数据商店(在iTop标准数据模型中不存在)。在使用TeemIp(模块或独立)的情况下,可以收集逻辑接口。

如果启用IP收集,则仅收集IPv4地址,而不收集IPv6。

采集器的当前版本旨在从单个vSphere服务器收集信息。要将信息表单和多台vSphere服务器收集和协调到一个iTop中,请选中从多个vSphere服务器收集下面。

要求

  • PHP版本5.3.0(vSphere API库需要支持名称空间)
  • 访问vSphere Web Services API
  • 访问iTop Web服务(REST +同步_导入。php和同步_exec.php)
  • 基础数据采集器要求。

安装

  • 在将运行采集器应用的计算机上的文件夹中,展开zip归档的内容。该计算机必须可以通过Web访问vSphere服务器和iTop服务器。
  • 编辑文件conffparams.local.xml的内容以适合您的安装,并提供适当的凭据以连接到vSphere和iTop。

配置

例如,使用以下XML内容在conf目录中创建文件params.local.xml:

<?xml version="1.0" encoding="UTF-8"?>
<parameters>
  <itop_url>https://localhost/</itop_url>
  <itop_login>admin</itop_login>
  <itop_password>admin</itop_password>
  <vsphere_uri>192.168.10.12:443</vsphere_uri>
  <vsphere_login>admin</vsphere_login>
  <vsphere_password>admin</vsphere_password>
  <contact_to_notify>john.doe@demo.com</contact_to_notify>
  <synchro_user>admin</synchro_user>
  <default_org_id>Demo</default_org_id>
</parameters>
参数含义样品价值
itop_url指向iTop应用的URLhttps://localhost/itop
itop_login连接到iTop的登录名(用户账号)。必须具有管理员权利才能执行数据同步。管理员
itop_passwordiTop账号的密码。 
vsphere_uri连接到vSphere的地址运动。格式:<名称>:<端口>或<ip_address>:<端口>192.168.10.12:443
vsphere_login登录名用于连接到vSphere管理员
vsphere_password与vSphere登录名对应的密码 
vsphere_connection_options传递给VMWarephp库的PHP流上下文选项的列表,以便调优进行https连接。 
default_org_id将在其中创建配置项(服务器,系统管理程序,服务器场...)的默认组织的名称。演示版

若要绕过 SSL 证书验证(如果 vSphere 服务器使用 vSphere 安装的默认证书运行,则需要此验证)在 params.local.xml 文件中添加 folling 行:

 <vsphere_connection_options>
        <ssl>
                <verify_peer>0</verify_peer>
                <verify_peer_name>0</verify_peer_name>
                <allow_self_signed>1</allow_self_signed>
        </ssl>
  </vsphere_connection_options>

其他可选参数

可以重新定义以下参数以更改采集器的默认行为:

参数含义样品价值
max_chunk_size在一次迭代中处理的元素的最大数量(用于在 iTop 中上传和同步)。如果元素超过此数字,则进程将自动迭代。1000
itop_synchro_timeout等待执行一个数据同步任务的超时(以秒为单位) - php_curl600
stop_on_synchro_error在同步过程中发生错误时是否停止(是或否)。no
console_log_level控制台的输出级别。从 -1(无)到 9(调试)。6(info)
console_log_dateformat记录器时间格式[Y-m-d H:i:s]
curl_options使用 cUrl 连接到 iTop Web 服务时,可以在本节中指定 cUrl 选项。语法为 <CURLOPT_NAME_OF_THE_OPTION1>VALUE 1</CURLOPT_NAME_OF_THE_OPTION1>其中VALUE_x是:
选项的数值,
或相应的 PHP"定义"(大小写敏感)的字符串表示形式。
 
data_path可以定义多个php_curl,如下面的示例    
data_path 1.1.4 中的 New 存储收集器生成的临时文件的路径。您可以使用特殊占位符 %APPROOT% 来指定相对于收集器的根文件夹的 pth。
%APPROOT%/data

<curl_options>
    <CURLOPT_SSL_VERIFYHOST>0</CURLOPT_SSL_VERIFYHOST>
    <CURLOPT_SSL_VERIFYPEER>1</CURLOPT_SSL_VERIFYPEER>
  </curl_options>

测试 iTop REST API 连接。

您可能会遇到网络/身份验证问题,无法到达需要同步的 iTop 服务器。要测试该连接,请使用以下命令:

php toolkit/testconnection.php 
UNIX system
    curl_init exists: 1
Problem opening URL: https://localhost/iTop/webservices/rest.php?version=1.0
    error msg: Failed to connect to localhost port 443: Connection refused
    curl_init error code: 7 (cf https://www.php.net/manual/en/function.curl-errno.php)

数据源的配置中的占位符

用于配置数据源的JSON文件包含从以上配置初始化的几个占位符($contact_to_notify$),以及特定于数据源的其他占位符。这些占位符可以在参数文件的<json_placeholders>数据内部进行配置这些占位符::

<?xml version="1.0" encoding="UTF-8"?>
  <parameters>
    ...
    <json_placeholders type="hash">
      <prefix>vSphere</prefix>
      <full_load_interval>60</full_load_interval>
    </json_placeholders> 
    ...
  </parameters>
参数含义默认价值
synchro_user如果用于运行此同步的用户帐户不是管理员,则必须在此指定其登录名,因为 iTop 只允许管理员和指定的用户运行同步。 
contact_to_notifyiTop 中现有联系人的电子邮件地址,用于通知同步结果    
full_load_interval数据两个完全导入之间的延迟(以秒表示)。
 

full_load_interval

两次完整导入数据之间的延迟(以秒为单位)。如果采集器在超过此间隔的时间间隔内未检测到对象,则将其视为过时并在iTop中进行标记。调整此价值依赖的计划周期。604800
prefixiTop中所有Synchronization数据Sources名称的前缀。如果您运行采集器的多个实例(以从多个vSphere服务器收集信息),请变更此价值,以便每个数据源具有唯一的名称。vSphere

品牌,型号,OS家族和OS版本规范化

为品牌,型号和OS系列收集的值可能会有很大差异,因为它们来自非同类来源(BIOS,运行系统。..)。数据采集器包含一种简单的机制,可以在将它们导入iTop之前通过过滤器手册标准化这些值。

创建映射表是一个迭代的流程,因为当采集器遇到新值(具有不同BIOS的新计算机,新运行的系统…)时,必须对表进行调整。

映射表是在配置文件params.local.xml中定义和维护的(您可以从params.distrib.xml复制示例定义并随意调整)。brand_mapping 用于过滤品牌值,model_mapping用于模型,os_family_mapping用于OS系列,os_version_mapping 用于OS版本。

映射表只是模式和替换字符串的有序列表。对于列表中的每个条目,如果提供的价值与模式匹配,则将其替换为替换字符串。从第一个(列表的顶部)到最后一个(列表的底部)处理模式。在第一个成功匹配后,处理停止。

映射表中每个条目的格式为:

<pattern>/regular_expression/replacement_string</pattern>

哪里:

  • normal_表达式是一个PCRE常规表达式与输入价值匹配,
  • 如果输入价值与常规价值匹配,则replace_string是生成的价值。
  • 注意:任何字符都可以用作常规表达式的定界符,不仅限于此。但是分隔符既不能出现在表达式本身中,也不能出现在替换字符串中,否则结果将不确定。

替换字符串是要使用的文字价值,但表单中的占位符除外,%1 $ s,%2 $ s…占位符%1 $ s对应于输入字符串中与整个常规表达式匹配的部分。占位符%2 $ s,%3 $ s与常规表达式中的匹配组(即括号)相对应。

品牌映射表示例:

<brand_mapping type="array">
    <!-- Syntax /pattern/replacement where:
      any delimiter can be used (not only /) 
      but the delimiter cannot be present in the "replacement" string
      pattern is a RegExpr pattern
      replacement is a sprintf string in which:
          %1$s will be replaced by the whole matched text,
          %2$s will be replaced by the first matched group, if any group is defined in the RegExpr
          %3$s will be replaced by the second matched group, etc...
    -->
    <pattern>/IBM/IBM</pattern>
    <pattern>/Hewlett Packard/Hewlett-Packard</pattern>
    <pattern>/Hewlett-Packard/Hewlett-Packard</pattern>
    <pattern>/Dell/Dell</pattern>
    <pattern>/.*/%1$s</pattern>
  </brand_mapping>

这样的映射表的结果将是:

  • 包含字母IBM(区分大小写)的任何品牌都将成为IBM
  • 包含Hewlett Packard(区分大小写)的任何品牌都将变为Hewlett-Packard(请注意Hewlett和Packard之间的附加破折号)
  • 任何包含Hewlett-Packard(区分大小写)的品牌都将完全变成Hewlett-Packard
  • 包含字母Dell(区分大小写)的任何品牌都将变成完全Dell

确保您保留列表中的最后一项(匹配所有项),以避免丢失某些值。

IP和逻辑接口集合

采集器自动检测远程iTop应用上是否存在TeemIp(作为模块,甚至作为独立的应用)。如果是这种情况,则以下参数可能是应用IP地址和逻辑接口集合。

<?xml version="1.0" encoding="UTF-8"?>
  <parameters>
    ...
    <teemip_options type="hash">
      <collect_ips>yes</collect_ips>
      <default_ip_status>allocated</default_ip_status>
      <manage_ipv6>no</manage_ipv6>
      <manage_logical_interfaces>yes</manage_logical_interfaces>
      </teemip_options>
    ...
  </parameters>
参数含义样品价值
collect_ips触发IP地址收集
default_ip_status新创建的IP地址的状态已分配
manage_ipv6触发IPv6收集-尚无法使用没有
manage_logical_interfaces触发逻辑接口收集以及地址接口链接

在此阶段,不收集IPv6。这是由于采集器基座的局限性所致,目前尚无法处理此类对象。

可配置的收集器

从应用的版本1.0.12开始,可以通过在XMl参数文件中添加额外的定义来配置服务器和虚拟化主机收集器。

配置包含两个部分:

  1. 第一部分(在<json>标签下)允许更改同步数据Source的定义(通常由与采集器类关联的.json文件修复)。这对于同步默认情况下未同步的新字段或将字段上的对帐规则同步到变更等很有用。可以通过在配置中指定新的价值来修改JSON文件属性_list部分下的任何字段。
  2. 第二部分(<source>标签)定义了为属性收集数据的方法。该标签必须包含相对于虚拟化主机对象的(部分)PHP表达式(实际上是主机系统在vSphere模型中)

配置的两个部分都可以独立使用(一个只能更改JSON或仅更改集合,或两者都可以更改)。

以下示例在服务器上配置了“序列号”的集合(来自otherIdentifyingInfo [EnclosureSerialNumberTag]),还使用收集的序列号作为管理程序和物理服务器之间的协调密钥(字段服务器_id)。

config.local.xml
 
<?xml version="1.0" encoding="UTF-8"?>
<parameters>
 
  ...
 
  <custom_synchro>
    <vSphereHypervisorCollector>
      <fields>
        <server_id>
          <source>hardware->systemInfo->otherIdentifyingInfo[EnclosureSerialNumberTag]</source>
          <json>
            <reconciliation_attcode>serialnumber</reconciliation_attcode>
          </json>
        </server_id>
      </fields>
    </vSphereHypervisorCollector>
    <vSphereServerCollector>
      <fields>
        <serialnumber>
          <source>hardware->systemInfo->otherIdentifyingInfo[EnclosureSerialNumberTag]</source>
        </serialnumber>
      </fields>
    </vSphereServerCollector>
  </custom_synchro>
 
  ...
 
</parameters>

当前仅在otherIdentifyingInfo属性上支持在<source>标签中带有括号的特殊语法。

用法

命令行的执行将:

  1. 连接到iTop以创建同步数据源(或检查它们的定义是否已存在,并在需要时进行更新)
  2. 连接到vSphere以收集有关虚拟机管理程序,服务器场,虚拟机的信息
  3. 将收集的数据上传到iTop
  4. 动态将收集的数据与现有的iTop配置项一起使用。
要启动数据集合并与iTop同步,请运行以下命令(从安装应用的根目录):
php exec.php

以下(可选)命令行选项可用:

选项含义默认价值
- config_file指定配置文件的完整路径。如果省略此参数,则默认使用conffparams.local.xml文件。空的
--console_log_level = <leve>控制台的输出级别。从-1(无)到9(调试)。6(info)
- collect_only仅运行数据集合,但不运行带有iTop的动态数据
--synchro_only将以前收集的数据与iTop同步(存储在数据目录中)。不要运行该集合。
- configure_only检查(并根据需要更新)iTop中的同步数据源并退出。不要运行收集或同步 
--max_chunk_size = <size>一遍传递给流程的最大项目数,用于保留系统的内存。如果流程还有更多项目,则流程将进行迭代。1000
--help使用模式以显示 exec.php 帮助。 

排程

交互式运行数据收集器后,下一步是安排其执行,以便收集和导入定期自动执行。

数据收集器不提供任何特定的调度机制,但简单的命令行 php exec.php 可以使用 cron(在 Linux 系统上)或使用 Windows 上的任务调度程序进行调度。

为了获得最佳效果,不要忘记调整 (full_load_interval 节) 中的配置参数json_placeholders使其与调度频率一致。

运行收集器的几个实例

在许多情况下,使用一组不同的参数多次运行收集器可能很有用。例如,从多个 LDAP 服务器(用于 LDAP 的 iTop 数据收集器)收集人员信息,或从多个 vSphere 服务器收集虚拟机信息(用于 vSphere 的 iTop 数据收集器)。

在框架版本 1.1.4 之前,您必须完全复制收集器应用程序,并调整每个副本上的文件 conf/params.local.xml。

自版本 1.1.4 起,您可以只对收集器应用程序进行一次复制,并为每个要运行的集合指定不同的配置文件(使用命令行选项 --config_file)(即每个 LDAP 或 vSphere 服务器一个配置文件)。

但是,为了避免在收集数据和与 iTop 同步期间出现任何问题,必须在配置文件内正确配置以下参数:

在每个不同的<prefix>使用不同的配置文件中使用不同的配置文件。这可确保为每个配置文件创建一组特定的同步数据源。

对每个<data_path>不同的更改。这将导致收集器将其收集的所有数据(包括一些临时文件)存储在专用目录中。这可防止收集器的一个实例覆盖另一个实例的数据。可以使用语法或  <data_path>%APPROOT%/data/collector1</data> 在数据文件夹中创建子文件夹收集器1。

故障排除

如果在连接到vSphere服务器时遇到问题,请尝试以下故障排除步骤:

1.检查从运行PHP的系统到vSphere服务器的连接是否确实可行。使用wget之类的命令行工具连接到vSphere服务器。例如,如果您的vsphere_uri是192.168.10.12:443,请尝试:

wget --no-check-certificate -O - https://192.168.10.12:443

如果连接失败,则可能是防火墙阻止了您的请求,或者已将vSphere配置为在其他端口上运行(例如,9443)...请向您的IT部门咨询。

1.一旦连接看起来很好,就在conf/params.local.xml中配置适当的vsphere_uri价值,然后从命令行(collectors子目录中)启动:

php check_soap.php

预期的输出(带有vsphere_uri = 192.168.10.12:443)为:

连接到https://192.168.10.12:443/sdk

好的,回复看起来像一个有效的SOAP回复。

--------------------- DEBUG ----------------
The request returned:
<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
 xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Body>
<RetrieveServiceContentResponse xmlns="urn:vim25"><returnval><rootFolder type="Folder">group-d1</rootFolder><propertyCollector type="PropertyCollector">propertyCollector</propertyCollector><viewManager type="ViewManager">ViewManager</viewManager><about><name>VMware vCenter Server</name><fullName>VMware vCenter Server 4.1.0 build-345043</fullName><vendor>VMware, Inc.</vendor><version>4.1.0</version><build>345043</build><localeVersion>INTL</localeVersion><localeBuild>0</localeBuild><osType>win32-x86</osType><productLineId>vpx</productLineId><apiType>VirtualCenter</apiType><apiVersion>4.1</apiVersion></about><setting type="OptionManager">VpxSettings</setting><userDirectory type="UserDirectory">UserDirectory</userDirectory><sessionManager type="SessionManager">SessionManager</sessionManager><authorizationManager type="AuthorizationManager">AuthorizationManager</authorizationManager><perfManager type="PerformanceManager">PerfMgr</perfManager><scheduledTaskManager type="ScheduledTaskManager">ScheduledTaskManager</scheduledTaskManager><alarmManager type="AlarmManager">AlarmManager</alarmManager><eventManager type="EventManager">EventManager</eventManager><taskManager type="TaskManager">TaskManager</taskManager><extensionManager type="ExtensionManager">ExtensionManager</extensionManager><customizationSpecManager type="CustomizationSpecManager">CustomizationSpecManager</customizationSpecManager><customFieldsManager type="CustomFieldsManager">CustomFieldsManager</customFieldsManager><diagnosticManager type="DiagnosticManager">DiagMgr</diagnosticManager><licenseManager type="LicenseManager">LicenseManager</licenseManager><searchIndex type="SearchIndex">SearchIndex</searchIndex><fileManager type="FileManager">FileManager</fileManager><virtualDiskManager type="VirtualDiskManager">VirtualDiskManager</virtualDiskManager></returnval></RetrieveServiceContentResponse>
</soapenv:Body>
</soapenv:Envelope>
---------------------------------------------

请求返回:

如果结果不是预期的结果,请检查您的vSphere服务器是否已正确配置为以HTTPS运行(应为默认设置)。

数据系列引用

服务器和虚拟机管理程序

在vSphere Web Services SDK中,虚拟机监控程序和物理服务器由相同的对象表示主机系统.

使用以下映射,将来自vSphere HostSystem对象的信息在iTop中导入到服务器对象中:

iTop中的字段vSphere中的来源
name主机系统→名称
0rg_id配置文件提供的常量价值
brand_id来自HostSystem→hardware→systemInfo→供应商的信息通过名为brand_mapping的映射表进行处理
model_id来自HostSystem→hardware→systemInfo→model的信息通过名为model_mapping的映射表进行处理
cpuHostSystem → hardware → cpuInfo → numCpuPackages
ramHostSystem → hardware → memorySize divided by (1024*1024)
osfamily_id来自HostSystem → config → product → name 的信息通过名为os_family_mapping的映射表进行处理
osversion_id来自HostSystem→config→product→fullName的信息通过名为os_version_mapping的映射表进行处理
status常量价值:活性
managementip_id如果激活了IP收集,则指向IPAddress对象的外键

使用以下映射,将来自vSphere HostSystem对象的信息在iTop中导入到虚拟化主机对象中:

iTop中的字段vSphere中的来源
nameHostSystem → name
org-id配置文件提供的常量价值
server_id

HostSystem → name

本节中使用的示例可配置的收集器显示了如何收集服务器的序列号。请注意,属性otherIdentifyingInfo中可用的信息似乎取决于硬件制造商和虚拟化主机的版本。在您的系统上进行测试修改后,再进行生产。

农场

在vSphere Web Services SDK中,以对象表示集群集群计算资源.

使用以下映射,将来自vSphere ClusterComputeResource对象的信息在iTop中导入到集群对象中:

iTop中的字段vSphere中的来源
nameClusterComputeResource→名称
org-id配置文件提供的常量价值

属于集群的管理程序列表是通过属性收集的:ClusterComputeResource→主机→名称。

虚拟机

在vSphere Web Services SDK中,以对象表示虚拟机虚拟机。使用以下映射,将来自vSphere VirtualMachine对象的信息在iTop中导入到VirtualMachine对象中:

iTop中的字段vSphere中的来源
nameVirtualMachine→name
org-id配置文件提供的常量价值
description

VirtualMachine → config → annotation

cpuVirtualMachine → config → hardware → numCPU
ram

VirtualMachine → config → hardware → memoryMB 

virtualhost_id集群(如果不为空)或VirtualMachine→运行时→主机→名称
如果不使用TeemIp
管理ipVirtualMachine → guest → ipAddress
是否使用TeemIp IS和是否收集IP
managementip_idip设置为VirtualMachine→guest→ipAddress的IPAddress对象的外键

IPv4地址

如果启用(需要TeemIp),则使用以下映射将来自vSphere IPv4地址的信息在iTop中导入到IPv4Address对象中:

iTop中的字段vSphere中的来源
org-id配置文件提供的常量价值
status配置文件提供的常量价值
short_nameVirtualMachine → guest → ipAddress
ip地址虚拟机:VirtualMachine→guest →ipAddress
虚拟化主机:HostSystem → name

逻辑接口

如果启用(需要TeemIp),则采集器可以注册连接到虚拟机的逻辑接口。使用以下映射在iTop中将它们导入到LogicalInterface对象中:

iTop中的字段vSphere中的来源
MAC地址

VirtualMachine → guest → net → macAddress name

VirtualMachine → config → hardware → device → backing → 

name

* network → name or

* opaqueNetworkId or
* deviceName or
* port

virtualmachine_id

VirtualMachine→namr
ip_list接口附加的IP地址列表

ip_list属性通过专用的同步数据源vSphere:lnkIPInterfaceToIPAddress进行同步。

原文:https://www.itophub.io/wiki/page?id=extensions%3Avsphere-data-collector


Data collector for vSphere

name:
Data collector for vSphere
description:
Collector for vSphere data synchronization in iTop CMDB (VM, Hypervisor, Cluster)
version:
1.0.12
release:
2020-04-02
code:
combodo-data-collector-for-vsphere
state:
stable
alternate-name:
vSphere Data Collector
diffusion:
iTop Hub

This stand-alone application collects information about a whole data center from a vSphere server (using the vSphere Web Services) and synchronizes this information with iTop using several Synchronization Data Sources.

vSphere Data Collector connections

Features

  • Automated inventory of Servers (with their Brand, Model), Hypervisors (with their OS Family and OS Version), Farms, Virtual Machines (with their OSFamily and OS Version).

  • Optional inventory of IPv4 addresses and logical interfaces

  • The collector can reside on any system with web access to both vSphere Web Services and iTop.

  • Automatic creation and update of the Synchronization Data Sources in iTop.

  • Starting with version 1.0.12 the collection mechanism is somehow extensible

This collector makes use of iTop's built-in Data Synchronization mechanism. For more information about how the data synchronization works, refer to Data Synchronization Overviewand relies on Data collector Base mechanism

Revision History

Release DateVersionComments
2020-07-071.0.13- Differenciates error/logs between teemip NOT installed and itop REST API issue
- Multi configuration file
- New CSV collector
- Configurable timestamp added in the logs
- New option for usage: –help
2020-04-021.0.12- Made the Server and Hypervisor collectors configurable
- Fix for a crash caused by a blank datastore name (thanks to David Wißen from ITOMIG for reporting it).
- Fix for some OpenVM Tools reporting IP addresses with a trailing space (thanks to Martin Raenker from ITOMIG).
2018-12-311.0.11Corrects regression introduced by 1.0.10 :
- Improved support of iTop 2.4+ (obsolescence flag)
- Removed a debug trace
- Fix Virtual machines which “connectionState” is not “connected” are skipped
  2018-08-241.0.10Handles TeemIp :
- Automatically detects if TeemIp (as an iTop module or as a standalone application) is present.
- Optional synchronization of IPv4 addresses
- Optional synchronisation of logical interfaces
2018-07-021.0.9Robustness :
- more traces in debug mode (–console_log_level=9)
- skipping virtual machines which “connectionState” is not “connected”, since most of their properties are not accessible and would crash PHP/Soap (with a core dump !!)
2018-02-221.0.8- The os_version_mapping was used only for Hypervisors, made it work for VirtualMachines also.
Proper support of UTF-8 characters in regular expressions (the code now always uses the u modifier)
2017-04-241.0.7Fixed the value for the status of VMs: 'production' instead of 'active'.
2017-03-081.0.6- Code cleanup.
- Data Synchro status and full_load_interval are now homogeneous and configurable parameters for all data sources.
- OS Version and OS Family have their own mapping table.
- Fix for the Hypervisor status value (now defaults to “production”).
- Addition of the check_soap.php script for troubleshooting.
- Protect against a PHP crash in case of a disconnected ESX.
- Do not collect an IPV6 as the management IP address of a virtual machine (since the field can only contain an IPV4). Many thanks to Andrew Armstrong for proposing this fix.
2016-12-081.0.5New options to bypass SSL certificate validation and automatic detection of invalid certificates.
2015-11-161.0.4Replace line-breaks by spaces inside the description of a VM. Fix the +1 on the number of vCPUs
2015-04-171.0.3Added the collection of the “description” field for a VM
2015-02-231.0.2Servers must be loaded before hypervisors
2015-01-071.0.1Beta version
2014-05-131.0.0First alpha version

Limitations

This version of the collector does not collect the Data Stores (they do not exist in the iTop standard data model). Logical interfaces may be collected in the case where TeemIp (module or standalone) is used.

If IP collection is enabled, only IPv4 addresses are collected, not IPv6.

The current version of the collector is designed to collect information from a single vSphere server. To collect and reconcile information form several vSphere servers into one iTop, check the section Collecting from several vSphere servers below.

Requirements

  • PHP Version 5.3.0 (support of namespaces is required for the vSphere API library)

  • An access to the vSphere web services API

  • An access to the iTop web services (REST + synchro_import.php and synchro_exec.php)

  • Data collector Base requirements.

Installation

  • Expand the content of the zip archive on a folder on the machine that will run the collector application. This machine must have a web access to both the vSphere server and the iTop server.

  • Edit the content of the file conf/params.local.xml to suit your installation, supplying the appropriate credentials to connect to vSphere and iTop.

Configuration

Create the file params.local.xml in the conf directory with the following XML content:

<?xml version="1.0" encoding="UTF-8"?>
<parameters>
  <itop_url>https://localhost/</itop_url>
  <itop_login>admin</itop_login>
  <itop_password>admin</itop_password>
  <vsphere_uri>192.168.10.12:443</vsphere_uri>
  <vsphere_login>admin</vsphere_login>
  <vsphere_password>admin</vsphere_password>
  <contact_to_notify>john.doe@demo.com</contact_to_notify>
  <synchro_user>admin</synchro_user>
  <default_org_id>Demo</default_org_id>
</parameters>
ParameterMeaningSample value
itop_urlURL to the iTop Applicationhttps://localhost/itop
itop_loginLogin (user account) for connecting to iTop. Must have admin rights for executing the data synchro.admin
itop_passwordPassword for the iTop account. 
vsphere_uriThe address/port to connect to vSphere. Format: <name>:<port> or <ip_address>:<port>192.168.10.12:443
vsphere_loginLogin for connecting to vSphereadministrateur
vsphere_passwordPassword corresponding to the vSphere login 
vsphere_connection_optionsList of PHP Stream context options to pass to the VMWarephp library in order to tune the https connection. 
contact_to_notifyThe email address of an existing contact in iTop, to be notified of the results of the synchronization
default_org_idThe name of the default Organization in which the CIs (Servers, Hypervisors, Farms…) will be createdDemo

Other optional parameters

ParameterMeaningSample value
synchro_userIf the user account used for running this synchronization is not an Administrator, then its login must be specified here, since iTop allows only the administrators and the specified user to run the synchronization. 

Placeholders in the configuration of the data sources

The JSON files used to configure the data sources contain several placeholders initialized from the configuration above ($contact_to_notify$), but also additional placeholders specific to the data sources. These placeholders can be configured inside the <json_placeholders> tag in the parameters file:

<?xml version="1.0" encoding="UTF-8"?>
  <parameters>
    ...
    <json_placeholders type="hash">
      <prefix>vSphere</prefix>
      <full_load_interval>60</full_load_interval>
    </json_placeholders> 
    ...
  </parameters>
ParameterMeaningSample value
full_load_intervalThe delay (expressed in seconds) between two complete imports of the data. The objects which have not been detected by the collector during a timespan longer than this interval will be considered as obsolete and marked as such in iTop. Adjust this value depending on the scheduling recurrence.604800
prefixThe prefix for the name of all Synchronization Data Sources in iTop. If you run several instances of the collector (to collect information from several vSphere servers), change this value so that each data source has a unique namevSphere

The file params.distrib.xml contains the default values for the parameters. Both files (params.distrib.xml and params.local.xml) use exactly the same format. But params.distrib.xml is considered as the reference and should remain unmodified. Should you need to change the value of a parameter, copy and modify its definition in params.local.xml. The values inparams.local.xml have precedence over the ones in params.distrib.xml

To bypass SSL certificate validation (which is needed if your vSphere server runs with the default certificates installed by vSphere) add the folling lines in the params.local.xml file:

  <vsphere_connection_options>
        <ssl>
                <verify_peer>0</verify_peer>
                <verify_peer_name>0</verify_peer_name>
                <allow_self_signed>1</allow_self_signed>
        </ssl>
  </vsphere_connection_options>

Brands, Models, OS Family and OS Version normalization

The values collected for Brands, Models and OS Family may be very variable because they came from non homogeneous sources (BIOS, operating system…). The data collector contains a simple mechanism to filter / normalize the values via a manual tuning, before importing them into iTop: a simple mapping table.

Creating the Mapping Table is an iterative process, since the table must be adapted when the collector encounters new values (new machine with a different BIOS, new operating system…)

Mapping tables are defined and maintained in the configuration file params.local.xml (you can copy an example definition from params.distrib.xml and ajust it at your will). brand_mapping is used for filtering the Brand values, model_mapping is used for Models, os_family_mapping is used for OS Families andos_version_mapping is used for OS Versions.

mapping table is simply an ordered list of patterns and replacement strings. For each entry in the list, if the supplied value matches the pattern, it is replaced by the replacement string. The patterns are processed from the first one (top of the list) to the last one (bottom of the list). The processing stops after the first successful match.

The format of each entry in the mapping table is:

<pattern>/regular_expression/replacement_string</pattern>

Where:

  • regular_expression is a PCRE regular expression to be matched against the input value,

  • replacement_string is the resulting value if the input value matches the regular expression.

  • Note: any character can be use as a delimiter around the regular expression, not only /. But the delimiter character can be present neither in the expression itself nor in the replacement string or the result will be undetermined.

The replacement string is the literal value to use, with the exception of placeholders in the form %1$s, %2$s… The placeholder %1$s corresponds to the part of the input string that matches the whole regular expression. Placeholders %2$s, %3$s corresponds to matching groups (i.e. parentheses) in the regular expression.

Example of a mapping table for the brands:

  <brand_mapping type="array">
    <!-- Syntax /pattern/replacement where:
      any delimiter can be used (not only /) 
      but the delimiter cannot be present in the "replacement" string
      pattern is a RegExpr pattern
      replacement is a sprintf string in which:
          %1$s will be replaced by the whole matched text,
          %2$s will be replaced by the first matched group, if any group is defined in the RegExpr
          %3$s will be replaced by the second matched group, etc...
    -->
    <pattern>/IBM/IBM</pattern>
    <pattern>/Hewlett Packard/Hewlett-Packard</pattern>
    <pattern>/Hewlett-Packard/Hewlett-Packard</pattern>
    <pattern>/Dell/Dell</pattern>
    <pattern>/.*/%1$s</pattern>
  </brand_mapping>

The result of such a mapping table will be:

  • Any brand containing the letters IBM (case sensitive) will become exactly IBM

  • Any brand containing Hewlett Packard (case sensitive) will become exactly Hewlett-Packard (notice the added dash between Hewlett and Packard)

  • Any brand containing Hewlett-Packard (case sensitive) will become exactly Hewlett-Packard

  • Any brand containing the letters Dell (case sensitive) will become exactly Dell

Make sure that you keep the last (match all) item in the list to avoid loosing some values.

IPs and logical interfaces collection

The collector automatically detects if TeemIp is present on the remote iTop application (as a module or even as a stand-alone application). Should that be the case, then the following parameters may trigger IP addresses and logical interfaces collection.

<?xml version="1.0" encoding="UTF-8"?>
  <parameters>
    ...
    <teemip_options type="hash">
      <collect_ips>yes</collect_ips>
      <default_ip_status>allocated</default_ip_status>
      <manage_ipv6>no</manage_ipv6>
      <manage_logical_interfaces>yes</manage_logical_interfaces>
      </teemip_options>
    ...
  </parameters>
ParameterMeaningSample value
collect_ipsTriggers IP addresses collectionyes
default_ip_statusSatus of newly created IP addressesallocated
manage_ipv6Triggers IPv6 collection - not working yetno
manage_logical_interfacesTriggers logical interfaces collection as well as the addresses / interfaces linksyes

At this stage, IPv6 are not collected. This is due to a limitation of the collector base which cannot handle such objects yet.

Configurable collectors

Starting with version 1.0.12 of the application, Server and Hypervisor collectors are configurable by adding extra definitions in the XMl parameter file.

This configuration contains two parts:

  1. The first part (under the <json> tag) allows to alter the definition of the Synchro Data Source (normally fixed by the .json file associated with the collector class). This is useful to synchronise a new field which was not synchronized by default, or to change a reconciliation rule on a field, etc… Any field under the attribute_list part of the JSON file can be modified by specifying a new value in the configuration.

  2. The second part (the <source> tag) defines the way to collect the data for the attribute. This tag must contain a (partial) PHP expression relative to the Hypervisor object (which is actually a HostSystem in the vSphere model)

Both parts of the configuration can be used independently (one can alter only the JSON or only the collection, or both).

The example below configures the collection of the “Serial Number” on Servers (from the otherIdentifyingInfo[EnclosureSerialNumberTag]) and also uses the collected serial number as the reconciliation key between Hypervisors and physical Servers (the field server_id).

config.local.xml
 
<?xml version="1.0" encoding="UTF-8"?>
<parameters>
 
  ...
 
  <custom_synchro>
    <vSphereHypervisorCollector>
      <fields>
        <server_id>
          <source>hardware->systemInfo->otherIdentifyingInfo[EnclosureSerialNumberTag]</source>
          <json>
            <reconciliation_attcode>serialnumber</reconciliation_attcode>
          </json>
        </server_id>
      </fields>
    </vSphereHypervisorCollector>
    <vSphereServerCollector>
      <fields>
        <serialnumber>
          <source>hardware->systemInfo->otherIdentifyingInfo[EnclosureSerialNumberTag]</source>
        </serialnumber>
      </fields>
    </vSphereServerCollector>
  </custom_synchro>
 
  ...
 
</parameters>

The special syntax with brackets in the <source> tag is currently only supported on the otherIdentifyingInfo property.

Usage

To launch the data collection and synchronization with iTop, run the following command (from the root directory where the data collector application is installed):

php exec.php

The following (optional) command line options are available:

OptionMeaningdefault value
--console_log_level=<level>Level of output to the console. From -1 (none) to 9 (debug).6 (info)
--collect_onlyRun only the data collection, but do not synchronize the data with iTopfalse
--synchro_onlySynchronizes the data previously collected (stored in the data directory) with iTop. Do not run the collection.false
--configure_onlyCheck (and update if necessary) the synchronization data sources in iTop and exit. Do NOT run the collection or the synchronization 
--max_chunk_size=<size>Maximum number of items to process in one pass, for preserving the memory of the system. If there are more items to process, the application will iterate.1000

The execution of the command line will:

  1. Connect to iTop to create the Synchronization Data Sources (or check their definition if they already exist, updating them if needed)

  2. Connect to vSphere to collect the information about the Hypervisors, Farms, Virtual Machines

  3. Upload the collected data into iTop

  4. Synchronize the collected data with the existing iTop CIs.

Troubleshooting

If you have troubles connecting to the vSphere server, try the following troubleshooting steps:

  1. Check that the connection from the system running PHP to the vSphere server is actually possible. Use a command line tool like wget to connect to the vSphere server. For example if your vsphere_uri is 192.168.10.12:443, try:
wget --no-check-certificate -O - https://192.168.10.12:443

If the connection does not succeed, there may be a firewall blocking your requests, or that vSphere is condifured to operate on a different port (for example 9443)… ask your IT department about it.

  1. Once the connection seem to go fine, configure the proper vsphere_uri value in conf/params.local.xml and, from the command line (in the collectorssubdirectory) launch:
php check_soap.php

The expected output (with vsphere_uri = 192.168.10.12:443) is:

Connecting to https://192.168.10.12:443/sdk

Ok, the response looks like a valid SOAP response.

--------------------- DEBUG ----------------
The request returned:
<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
 xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Body>
<RetrieveServiceContentResponse xmlns="urn:vim25"><returnval><rootFolder type="Folder">group-d1</rootFolder><propertyCollector type="PropertyCollector">propertyCollector</propertyCollector><viewManager type="ViewManager">ViewManager</viewManager><about><name>VMware vCenter Server</name><fullName>VMware vCenter Server 4.1.0 build-345043</fullName><vendor>VMware, Inc.</vendor><version>4.1.0</version><build>345043</build><localeVersion>INTL</localeVersion><localeBuild>0</localeBuild><osType>win32-x86</osType><productLineId>vpx</productLineId><apiType>VirtualCenter</apiType><apiVersion>4.1</apiVersion></about><setting type="OptionManager">VpxSettings</setting><userDirectory type="UserDirectory">UserDirectory</userDirectory><sessionManager type="SessionManager">SessionManager</sessionManager><authorizationManager type="AuthorizationManager">AuthorizationManager</authorizationManager><perfManager type="PerformanceManager">PerfMgr</perfManager><scheduledTaskManager type="ScheduledTaskManager">ScheduledTaskManager</scheduledTaskManager><alarmManager type="AlarmManager">AlarmManager</alarmManager><eventManager type="EventManager">EventManager</eventManager><taskManager type="TaskManager">TaskManager</taskManager><extensionManager type="ExtensionManager">ExtensionManager</extensionManager><customizationSpecManager type="CustomizationSpecManager">CustomizationSpecManager</customizationSpecManager><customFieldsManager type="CustomFieldsManager">CustomFieldsManager</customFieldsManager><diagnosticManager type="DiagnosticManager">DiagMgr</diagnosticManager><licenseManager type="LicenseManager">LicenseManager</licenseManager><searchIndex type="SearchIndex">SearchIndex</searchIndex><fileManager type="FileManager">FileManager</fileManager><virtualDiskManager type="VirtualDiskManager">VirtualDiskManager</virtualDiskManager></returnval></RetrieveServiceContentResponse>
</soapenv:Body>
</soapenv:Envelope>
---------------------------------------------

If the result is not the expected one, check that your vSphere server is properly configured to run as HTTPS (which should be the default).

Scheduling

Once you've run the data collector interactively, the next step is to schedule its execution so that the collection and import occurs automatically at regular intervals.

The data collector does not provide any specific scheduling mechanism, but the simple command line php exec.php can be scheduled with either cron (on Linux systems) or using the Task Scheduler on Windows.

For optimal results, don't forget to adjust the configuration parameter full_load_interval in the (json_placeholders section) to make it consistent with the frequency of the scheduling.

Data Collection Reference

Servers and Hypervisors

In the vSphere web services SDK, Hypervisors and physical Servers are represented by the same object HostSystem.

The information from the vSphere HostSystem object is imported in iTop into the Server object using the following mapping:

Field in iTopSource in vSphere
nameHostSystem → name
org_idConstant value, supplied by the configuration file
brand_idThe information from HostSystem → hardware → systemInfo → vendor is processed through the mapping table named brand_mapping
model_idThe information from HostSystem → hardware → systemInfo → model is processed through the mapping table named model_mapping
cpuHostSystem → hardware → cpuInfo → numCpuPackages
ramHostSystem → hardware → memorySize divided by (1024*1024)
osfamily_idThe information from HostSystem → config → product → name is processed through the mapping table named os_family_mapping
osversion_idThe information from HostSystem → config → product → fullName is processed through the mapping table named os_version_mapping
statusConstant value: active
managementip_idForeign key to an IPAddress object, if IP collection is activated

The information from the vSphere HostSystem object is imported in iTop into the Hypervisor object using the following mapping:

Field in iTopSource in vSphere
nameHostSystem → name
org_idConstant value, supplied by the configuration file
server_idHostSystem → name

The example used in the section Configurable Collectors shows how to collect the serial number of servers. Be aware that the information available in the property otherIdentifyingInfo seem to depend on the hardware manufacturer and/or the version of the hypervisor. Test on your systems before putting such a modification in production.

Farms

In the vSphere web services SDK, a Farm is represented by the object ClusterComputeResource.

The information from the vSphere ClusterComputeResource object is imported in iTop into the Farm object using the following mapping:

Field in iTopSource in vSphere
nameClusterComputeResource → name
org_idConstant value, supplied by the configuration file

The list of hypervisors belonging to a farm is collected via the property: ClusterComputeResource → host → name.

Virtual Machines

In the vSphere web services SDK, a Virtual Machine is represented by the object VirtualMachine. The information from the vSphere VirtualMachine object is imported in iTop into the VirtualMachine object using the following mapping:

Field in iTopSource in vSphere
nameVirtualMachine → name
org_idConstant value, supplied by the configuration file
descriptionVirtualMachine → config → annotation
cpuVirtualMachine → config → hardware → numCPU
ramVirtualMachine → config → hardware → memoryMB
virtualhost_ideither the Farm (if not empty) or VirtualMachine → runtime → host → name
If TeemIp is NOT used
managementipVirtualMachine → guest → ipAddress
If TeemIp IS used and if IPs are collected
managementip_idForeign key to an IPAddress object with ip set to VirtualMachine → guest → ipAddress

IPv4 Addresses

If enabled (TeemIp required), the information from the vSphere IPv4 addresses is imported in iTop into the IPv4Address object using the following mapping:

Field in iTopSource in vSphere
org_idConstant value, supplied by the configuration file
statusConstant value, supplied by the configuration file
short_nameVirtualMachine → guest → hostName
ipVirtual Machine: VirtualMachine → guest → ipAddress
Hypervisor: HostSystem → name

Logical Interfaces

If enabled (TeemIp required), the collector may register the logical interfaces attached to virtual machines. These are imported in iTop into the LogicalInterface objects using the following mapping:

Field in iTopSource in vSphere
macaddressVirtualMachine → guest → net → macAddress
nameVirtualMachine → config → hardware → device → backing →
* network → name or
* opaqueNetworkId or
* deviceName or
* port
virtualmachine_idVirtualMachine → name
ip_listList of IP addresses attached to the interface

The ip_list attribute is synchronized through the dedicated synchro data source vSphere:lnkIPInterfaceToIPAddress.

标签:
由 superadmin 在 2020/08/27, 16:06 创建
    

需要帮助?

如果您需要有关XWiki的帮助,可以联系:

深圳市艾拓先锋企业管理咨询有限公司