基础数据收集器

名称:基础数据采集器

描述:库存数据采集器工具包用于为iTop创建自己的数据收藏家

版本:1.1.4

发布:2020-07-07

状态:稳定

扩散:客户商店,Combodo网站

该模块为创建用于iTop的工业数据集合和同步应用提供了基础。开发人员可以依靠此模块来执行与iTop数据导入和同步数据相关的所有繁重工作,以便专注于数据集合。

下载1.1.4

特征

  • 简单的API,使创建新的采集器变得容易快捷。
  • 根据JSON定义在iTop中自动创建和更新Synchronization数据源。
  • 通过明确的“可选”属性,支持对目标数据模型的细微变化
  • CSV格式的基本验证与同步数据源中的预期字段进行了比较。
  • 数据通过max_chunk_size(可配置)的块进行上传和同步。
  • 用于处理参数配置的可扩展机制。
  • JSON定义中占位符的可扩展列表。
  • 通知联系人的自动管理(通过占位符)和用于运行同步的用户。
  • PHP的最小版本的验证和所需的扩展(由采集器指定)。
  • 命令行工具,用于从iTop中的现有同步数据源生成JSON定义。
  • 能力独立运行:数据源的配置,数据集合和数据同步。
  • 可配置的日志级别(用于控制台输出或syslog日志记录)
  • 快速创建基于SQL的收集器的简单框架

修订记录

发布日期版本注释
2020-10-281.2.0

* 新的JSON收集器.
* CSV器配置:格式+新参数的变化 defaults / field / ignored_columns / has_header
* 添加testcon/ection. php脚本

* 配置文件的路径现在可以通过选项--config_file在命令指定线.

2020-06-241.1.4*现在可以通过命令行上的--config_file选项指定配置文件的路径。
*现在,存储收集到的数据的位置是配置文件中的参数:data_path。
*更好地检查数据源定义以捕获丢失的对帐键
* Lookuptable类的选项将查找错误视为正常错误
2020-04-301.1.3*新的CSV采集器
*在日志中添加了可配置的时间戳
*用法的新选项:–帮助
2019-11-071.1.2修复“未定义常量TABLENAME_PATTERN”
2019-10-281.1.1包含从1.0.13和1.1.0的升级
*拒绝database_table_name的无效字符
2019-10-281.1.0基于1.0.9
*添加了特定的类MySQLCollector,该类强制数据库连接使用UTF-8字符
2019-10-281.0.13* LookupTables现在可以不区分大小写了(因为MySQL不区分大小写)
*防止在SQLCollector中为每个“忽略的”属性警告
*改进了对iTop 2.4+(废弃标志)的支持
2019-10-281.0.12*删除了PHP 7.2中的警告
2018-06-261.0.11添加了调试跟踪(如果console_log_level = 9,则可见)以显示应用了常规表达式的映射(应用时)。
漏洞修复:正确处理映射表正则表达式中的utf-8字符((u修饰符)
使cUrllSSL选项可配置为适合所有可能的组合和安全注意事项。
2015-06-301.0.10采集器的新类:MySQLCollector,它强制将检索到的数据编码为U-8。
2015-06-091.0.9性能增强功能:构建查找表时仅检索所需的字段。
2015-06-021.0.8更好地检查文件访问权利以进行写入。 SQL连接字符串(用于SQL收集器)现在可以完全配置。
2015-05-201.0.7错误修复:支持在文件名中使用反斜杠。通过将Utils :: Substitute()标记为静态,删除了警告。
2015-05-131.0.6添加了在重新处理数据时“忽略”某些行的支持。可以将SQL采集器配置为安全地忽略某些字段。
2015-02-161.0.4添加了配置参数stop_on_同步_错误。
2015-01-061.0.3处理非U-8数据(通过GetCharset()的重载),错误检查数据数据阶段,iTop 2.1.0的优化:忽略变更_table_name字段中的任何导入。
2014-11-031.0.2添加了基类SQLCollector,可轻松创建基于SQL的收集器。
2014-10-111.0.1添加了AttributeIsOptional方法来处理目标数据模型中的变化。
2014-05-131.0.0第一个版本

局限性

  • 仅通过syncho_import Web服务将数据上传到iTop(可以使用命令行版本或直接使用SQL命令。也许稍后再使用)
  • 在SVN的iTop版本3805之前(从2015-10-12开始!),如果用于连接到iTop的账号未配置为使用英语作为语言,则采集器将无法正常工作!

要求

  • PHP版本5.3.0(某些收集器可能需要支持名称空间)
  • 访问iTop Web服务((REST + synchro_import.php and synchro_exec.php)
  • 我们建议安装php_curl以使用采集器基本参数 itop_synchro_timeout ,否则超时将被硬编码为200秒,并且不能被采集器覆盖。

在1000个同步对象下,如果没有php_curl,它应该可以工作,而在上面不能!

安装

  • 在将运行采集器的机器上的文件夹中,展开zip归档的内容。
  • 编辑文件conf/params.local.xml的内容以适合您的安装。

配置

原则

params.distrib.xml文件包含参数的默认值。这两个文件(params.distrib.xml和params.local.xml)使用完全相同的格式。但是,params.distrib.xml被视为引用,应保持不变。如果您需要变更的变更和价值,请复制并修改其定义inparams.local.xml。 params.local.xml中的值优先于params.distrib.xml中的值

 

配置

params.local.xml是唯一可编辑以配置采集器的文件。

至少必须在此文件中设置以下参数:

<itop_url>https://localhost/</itop_url> <itop_login>admin</itop_login> <itop_password>admin</itop_password>

参数含义样品价值
itop_login连接到iTop的登录名(用户账号)。必须具有管理员权利才能执行数据同步。管理员
itop_passwordiTop账号的密码。 
itop_urliTop应用的URLhttps://localhost/itop

可选参数

可以重新定义以下参数以更改采集器的默认行为:

参数含义默认价值
max_chunk_size一次迭代中流程的最大元素数(用于iTop中的上载和同步)。如果元素数超过此数目,则流程将自动进行迭代。1000
itop_synchro_timeout等待执行一个数据同步任务的超时(以秒为单位)-需要php_curl600
stop_on_synchro_error在同步期间发生错误时是否停止(是或否)。没有
console_log_level控制台的输出级别。从-1(无)到9(调试)。6(信息)
console_log_dateformat记录器时间戳格式[Y-m-d H:i:s]
curl_options使用cUrl连接到iTop Web服务时,可以在本节中指定cUrl选项。语法是<CURLOPT_NAME_OF_THE_OPTION1>价值_1 << CURLOPT_NAME_OF_THE_OPTION1>,其中价值_x是:
选项的数字价值,
或相应PHP“定义”的字符串表示形式(区分大小写)。
可以定义几个php_curl选项,如下例所示
 
数据_path1.1.4中的新增功能存储采集器生成的临时文件的路径。您可以使用特殊的占位符%APPROOT%来指定相对于采集器根文件夹的pth。%APPROOT %%数据
<curl_options>
    <CURLOPT_SSL_VERIFYHOST>0</CURLOPT_SSL_VERIFYHOST>
    <CURLOPT_SSL_VERIFYPEER>1</CURLOPT_SSL_VERIFYPEER>
  </curl_options>

数据源的配置中的占位符

用于配置数据源的JSON文件包含从以上配置初始化的几个占位符($联系人_to_notify $),以及特定于数据源的其他占位符。这些占位符可以在参数文件的<json_placeholders>数据内部进行配置:

参数含义默认价值
synchro_user如果用于运行此同步的用户账号不是管理员,则必须在此处指定其登录名,因为iTop仅允许管理员和指定的用户运行同步。 
contact_to_notifyiTop中现有联系人的发送邮件地址,将被通知同步结果 

full_load_interval

两次完整导入数据之间的延迟(以秒为单位)。如果采集器在超过此间隔的时间间隔内未检测到对象,则将其视为过时并在iTop中进行标记。调整此价值依赖的计划周期。604800
prefixiTop中所有Synchronization数据Sources名称的前缀。如果您运行采集器的多个实例(以从多个vSphere服务器收集信息),请变更此价值,以便每个数据源具有唯一的名称。 

用法

要启动数据集合并与iTop同步,请运行以下命令(从安装应用的根目录):

php exec.php

以下(可选)命令行选项可用:

选项含义默认价值
--config_file指定配置文件的完整路径。如果忽略此参数,则默认使用文件conf/params.local.xml。空的
--console_log_level = <level>控制台的输出级别。从-1(无)到9(调试)。6(信息)
--collect_only仅运行数据集合,但不运行带有iTop的动态数据
--synchro_only将先前收集的数据(存储在数据目录中)与iTop同步。不要运行该集合。
--configure_only在iTop中检查(并根据需要更新)同步数据源并退出。不要运行收集或同步 
--max_chunk_size = <size>一次保存流程的最大项目数,以保留系统的内存。如果流程还有更多项目,则流程将进行迭代。1000
-help使用模式显示exec.php帮助。 

运行采集器的多个实例

在许多情况下,使用一组不同的参数运行采集器几次可能会很有用。例如,从多个LDAP服务器(iTop LDAP数据采集器)收集人员信息,或从多个vSphere服务器(iTop vSphere数据采集器)收集虚拟机信息。

在框架版本1.1.4之前,您必须完全复制采集器应用并在每个副本上调整文件conf/params.local.xml。

从版本1.1.4开始,您只能拥有一份采集器应用的副本,并为要运行的每个集合指定一个不同的配置文件(使用命令行选项--config_file)(即每个LDAP或vSphere配置一个配置文件)。

但是,为避免在收集数据以及与iTop同步期间出现任何麻烦,必须在配置文件中正确配置以下参数:

  • 在每个不同的配置文件中使用不同的<prefix>。这样可以确保为每个配置文件创建一组特定的同步数据源。
  • 为每个配置文件使用不同的<数据_path>变量。这将导致采集器将其收集的所有数据(包括一些临时文件)存储在专用目录中。这样可以防止采集器的一个实例覆盖另一实例的数据。您可以使用数据<数据_path>%APPROOT %% data/collector1 <<数据>在数据文件夹中创建一个子文件夹collector1。

创建一个采集器

有关采集器的详细信息位于“收集器”文件夹中。此文件夹内必须至少有一个文件main.php。 main.php的目的是为模块注册所有采集器类,并加载相应的类(通过require_once(…)或通过注册自动加载器)。

采集器是一个PHP类,它为给定的同步数据源提供数据。采集器类是从抽象采集器类派生的。每个采集器与一个以JSON格式定义的同步数据源关联。默认的数据只是在collectors文件夹中查找与采集器类名称相同且扩展名为“ .json”的JSON文件。

指定所需的扩展

如果您的采集器需求是特定扩展名(或最低PHP版本),则可以通过在main.php中调用静态方法Orchestrator :: AddRequirement($ sMinRequiredVersion,$ sExtension ='PHP')来指示此依赖:

例如:

Orchestrator::AddRequirement('5.4.0'); //This requires at least PHP 5.4
Orchestrator::AddRequirement('1.2.0', 'ldap'); //This requires at least the ldap extension version 1.2.0

创建JSON定义文件

为同步数据源创建JSON文件的更简单方法是,对导出定义现有数据源。

  • 在iTop中创建同步数据源,并调整其参数(属性等)以适合您的需求
  • 使用命令行工具dump_tasks.php(在工具包文件夹中提供)来生成JSON文件:
php toolkit/dump_tasks.php --task_name="name of the task to export" > collectors/myCollector.json

在数据源定义中,可以使用特殊的占位符使数据源可以由应用的数据配置,或通过一些特殊设置来调整其行为:

占位符代码含义样品价值
$version$模块的版本。用于对应用进行版本控制,例如在同步数据源的“描述”中。1.0.0
$synchro_user$用户运行同步,由配置文件中的登录名指定。用户对象的标识符可通过此占位符获得。12
$contact_to_notify$由配置文件中的发送邮件地址指定的联系人进行通知。联系人的标识符通过此占位符提供。48

在上面列出的3个特殊参数配置之上,配置文件的“ json_placeholders”部分中定义的所有参数也可用作JSON文件内的占位符。

示例配置文件:

params.local.xml
 
<?xml version="1.0" encoding="UTF-8"?>
<!--  Local values for parameters. -->
<!--  The values defined in this file have precedence over the ones defined in params.distrib.xml -->
<parameters>
  <itop_url>https://localhost/trunk</itop_url>
  <itop_login>admin</itop_login>
  <itop_password>admin</itop_password>
  <console_log_level>9</console_log_level>
  <contact_to_notify>denis.flaven@combodo.com</contact_to_notify>
  <synchro_user>admin</synchro_user>
  <json_placeholders type="hash">
    <test>Test 1</test>
  </json_placeholders>
</parameters>

示例同步数据源定义文件,请注意$version$, $synchro_user$, $contact_to_notify$ and$test$ placeholders:的使用:

如果为database_table_name, this name MUST BEGIN WITH <table-prefix>synchro_data。其中<prefix>是iTop中所有表使用的前缀(使用iTop配置文件中的db_subname参数配置)。

如果修改已经收集的类的字段列表,则必须更新JSON定义文件,以指定如何处理新字段。默认情况下,它们被添加到现有数据同步中,没有更新且没有锁定。

实施采集器

您的采集器必须是从采集器派生的类。它必须(至少)实现Fetch()方法。对于每个要加载的对象,访存必须返回格式为属性_code =>价值的数组,或者在到达对象集的末尾时返回false。

Fetch()返回的数组必须包含:

  • 条目primary_key唯一标识正在与iTop同步的对象。该条目可以包含可以从库存集合中获得的任何唯一ID,也可以包含作为对象的各个字段的组合而生成的唯一标识符。由采集器应用来保证此标识符的唯一性(及其时间稳定性)
  • 要在iTop中加载的对象的每个属性的条目。

下面的示例代码生成了一组10个服务器,分别名为“服务器1”,“服务器2” ...“服务器10”,并初始化了服务器的3个字段:它们的名称,它们的组织(始终为“ Demo”)及其描述。

main.php
 
class MyCollector extends Collector
{
  protected $idx;
 
  public function Prepare()
  {
        $bResult = parent::Prepare();
        $this->idx = 0;
        return $bResult;
  }
 
  public function Fetch()
  {
     if ($this->idx < 10)
     {
        $this->idx++;
        return (
           'primary_key' => $this->idx, 
           'name' => 'Server '.$this->idx, 
           'org_id' => 'Demo', 
           'description' => 'Test Collector'
        );
     }
     return false;
  }
}
 
// Register the collector, as the 1st to run
Orchestrator::AddCollector(1, 'MyCollector');

如果采集器返回的数据未使用U-8字符集编码,请重载采集器的GetCharset()方法以返回字符集的名称(必须返回iTop采集器上iconv接受的价值)

注册您的采集器

要注册采集器,请调用静态方法Orchestrator :: AddCollector()。这两个参数是:

  • 采集器的运行顺序(当您需要一个接一个地运行多个收集器时)
  • 实现采集器的类的名称(从采集器派生)。

参数的默认值

采集器模块可以通过在collectors文件夹中提供文件params.distrib.xml来为其参数提供默认值。如果存在这样的文件,则将其值合并到conf目录中的等效文件上。

现有的基础收藏家

要创建新的采集器,可以将其基于标准或使用最近添加的那些收集器之一,这些收集器已经在数据来源上完成了依赖的工作:

CSV采集器

SQL采集器

JSON采集器(将于2021年发布)

SQL收集器

“核心”文件夹提供了一个抽象类SQLCollector,它可以用作快速创建通过SQL查询检索其数据的收集器的基础。

要创建这样的采集器,您需要:

  1. 创建从SQLCollector派生的类
  2. 为数据同步源创建json定义文件
  3. 添加配置参数(在params.distrib.xml中)以定义要运行的SQL查询
  4. 在collectors/main.php中注册您的采集器

用于SQL收集器的参数配置是:

参数含义默认价值
sql_enginePDO驱动引擎 用于数据库连接。MySQL的
sql_host要连接的数据库服务器的名称或IP地址。本地主机
sql_database要连接的数据库的名称。空的
sql_login连接到数据库时要使用的登录名
sql_password连接到数据库时使用的密码n/a
sql_connection_string1.0.8中的新增功能PDO连接字符串的格式。格式字符串内有3个占位符:%1 $ s = sql_engine,%2 $ s = sql_数据库和%3 $ s = sql_host%1 $ s:dbname =%2 $ s;host=%3 $ s
collector_class_query为采集器运行的查询,PHP类为collector_class 
collector_class_ignored_attribute1.0.6中的新增功能为了将数据模型的可能的变种纳入账号中,而不必每次都重写采集器,可以将某些收集的属性标记为“可选”,以便即使相应的采集器也可以运行采集器。在数据模型中不存在。供应在此忽略采集器代码数组。 

要指定端口号(或任何其他驱动程序特定的选项),请使用变更sql_connection_string的格式。例如:%1 $ s:dbname =%2 $ s; host =%3 $ s; port = 3307

对于1.0.8之前的版本,要指定端口号(默认端口除外),请使用语法主机; sql_host参数使用port = xxxx。例如:localhost; port = 3307

从版本1.0.10开始,该框架提供了采集器的新类:MySQLCollector。该类与SQLCollector相同,除了它通过在与数据库的每个连接的开始处发出SQL命令SET NAMES'u8'来强制将检索到的数据编码为U-8之外。为了避免任何带有数据字符集的问题,建议对与MySQLLMariaDB数据库的所有连接使用此新类。

简单SQL采集器的示例

让我们创建一个非常简单的SQL采集器,它将“笔记”文档(类DocumentNote)从一个iTop实例复制到另一个。由于采集器从基类继承了所有行为,因此采集器的PHP代码很简单:

DocumentNotesCollector.class.inc.php
 

<?php class DocumentNoteCollector extends SQLCollector { }

Find here a sample of an SQL collector definition file.

Then in params.distrib.xml, add the following entries:

  <sql_database>test</sql_database>
  <sql_login>root</sql_login>
  <sql_password>s3cret</sql_password>
  <documentnotecollector_query>SELECT id as primary_key, name, text, description, status, '2.0' as version, documenttype_id, 1 as org_id FROM view_DocumentNote</documentnotecollector_query>
  <documentnotecollector_ignored_attributes type="array">
    <attribute>location_id</attribute>
    <attribute>version_id</attribute>
  </documentnotecollector_ignored_attributes>

然后在params.distrib.xml中,添加以下条目:

main.php
 
<?php
require_once(APPROOT.'collectors/DocumentNoteCollector.class.inc.php');
Orchestrator::AddCollector(1 /* $iRank */, 'DocumentNoteCollector');

CSV采集器

“核心”文件夹提供了一个抽象类CSVCollector,它可作为快速创建从CSV文件检索其数据的收集器的基础。

要创建这样的采集器,您需要:

  1. 创建从CSVCollector派生的类
  2. 为数据同步源创建json定义文件
  3. 添加配置参数(在params.distrib.xml中)以定义要解析的CSV文件
  4. 在collectors/main.php中注册您的采集器

CSV收集器的参数配置是:

参数含义默认价值
collector_class_csv要为采集器解析哪个PHP类的csv文件。您可以指定此文件的完整路径((tmppmyfile.csv)或collector_class的相对路径。此采集器是必需的 
collector_class_commandCLI命令,用于在采集器解析csv文件之前执行该操作,PHP类是采集器_class。该采集器是可选的 
collector_class_encoding采集器的csv文件编码,其PHP类为collector_class。该采集器是可选的UTF-8
collector_class_separator用于csv文件解析的分隔符。该参数是可选的;

简单CSV采集器的示例

让我们创建一个非常简单的CSV采集器,它复制“ Person”对象(Person类)

由于采集器从基类继承了所有行为,因此采集器的PHP代码很简单:

iTopPersonCsvCollector.class.inc.php
 

<?php class iTopPersonCsvCollector extends CSVCollector { }

在这里找到一个收集器定义的示例CSV

然后在params.distrib.xml中,添加以下条目:

<?xml version="1.0" encoding="UTF-8"?>
<parameters>
  <iTopPersonCsvCollector_csv>collectors/iTopPersonCsvCollector.csv</iTopPersonCsvCollector_csv>
  <iTopPersonCsvCollector_command>sed -i -e 's|isaac|ISAAC|g' collectors/iTopPersonCsvCollector.csv</iTopPersonCsvCollector_command>
  <iTopPersonCsvCollector_encoding>UTF-8</iTopPersonCsvCollector_encoding>
</parameters>

最后,在collectors/main.php中添加以下行:

main.php
 
<?php
require_once(APPROOT.'collectors/iTopPersonCsvCollector.class.inc.php');
Orchestrator::AddCollector($index++, 'iTopPersonCsvCollector');

JSON采集器

“核心”文件夹提供了一个抽象类JSONCollector,它可以用作快速创建从JSON文件检索其数据的收集器的基础。

要创建这样的采集器,您需要:

  1. 创建从JSONCollector派生的类
  2. 为数据同步源创建json定义文件
  3. 添加一个配置参数(在params.distrib.xml中)以定义要解析的JSON文件
  4. 在collectors/main.php中注册您的采集器

JSON收集器的参数配置是:

参数含义
jsonflie定义要解析的采集器的json文件的相对或绝对路径,该PHP类是采集器collector_class。此采集器或collector_class_jsonurl是强制性的
jsonurl要为采集器解析哪个php类的json文件的URL。此参数或collector_class_jsonpath是必需的
jsonpost带有URL的参数Xml用来获取Json文件<name_of_param>价值<< name_of_param>
commandCLI命令,用于在采集器(其PHP类为collector_class)解析json文件之前执行。该采集器是可选的
pathjson分隔符中找到数据到动态的方法是,并且*用示例aaabb替换{“ aa”:{“ bb”:{mydata},“ cc”:“ xxx”}和aaa * bb替换任何单词aa“:{cc”:{“ bb”:{mydata1}},“ dd”:{“ bb”:{mydata2}}}
fieldsxml女巫描述json中的名称与itop中的名称之间的连接<name_in_json> name_in_itop << name_in_json>

一个简单的JSON采集器的示例

让我们创建一个非常简单的JSON采集器,它将“ Person”对象(类Person)从一个iTop复制到另一个

由于采集器从基类继承了所有行为,因此采集器的PHP代码很简单:

ITopPersonJsonCollector.class.inc.php
 

<?php class ITopPersonJsonCollector extends JsonCollector { }

JSON定义文件是

ITopPersonJsonCollector.json

然后在params.distrib.xml中,添加以下条目:

<?xml version="1.0" encoding="UTF-8"?>
<parameters>
        <itoppersonjsoncollector_jsonurl>http://localhost/iTop/webservices/rest.php</itoppersonjsoncollector_jsonurl>
        <itoppersonjsoncollector_jsonpost>
                <auth_user>restuser</auth_user>
                <auth_pwd>restuserpassword</auth_pwd>
                <json_data>{"operation": "core/get", "class": "Person", "key": "SELECT Person WHERE email LIKE '%.com'", "output_fields": "friendlyname, email, first_name, function, name, id, org_id,phone"}</json_data>
                <version>1.3</version>
        </itoppersonjsoncollector_jsonpost>
        <itoppersonjsoncollector_way>objects/*/fields</itoppersonjsoncollector_way>
        <itoppersonjsoncollector_fields>
                <primary_key>id</primary_key><!-- also this is not a field of the itop object, that column is mandatory -->
                <name>name</name>
                <status>status</status>
                <first_name>first_name</first_name>
                <email>email</email>
                <phone>phone</phone>
                <mobile_phone>mobile</mobile_phone>
                <function>function</function>
                <employee_number>employee_number</employee_number>
                <org_id>org_id</org_id>
        </itoppersonjsoncollector_fields>
        <itoppersonjsoncollector_defaults>
                <org_id>Demo</org_id>
                <status>active</status>
        </itoppersonjsoncollector_defaults>
        <json_placeholders>
                <!-- For compatibility with the version 1.1.x of the collector, define the data table names as following:
             <prefix></prefix>
             <persons_data_table>synchro_data_PersonAD</persons_data_table>
             <users_data_table></users_data_table>
              -->
                <prefix>$prefix$</prefix>
                <persons_data_table>synchro_data_person</persons_data_table>
        </json_placeholders>
</parameters>

最后,在collectors/main.php中添加以下行:

main.php
 
<?php
require_once(APPROOT.'collectors/ITopPersonJsonCollector.class.inc.php');
Orchestrator::AddCollector($index++, 'ITopPersonJsonCollector');

高级收藏家

采集器框架为现实的收集器提供了执行一些高级处理的方法:

  • 用于可配置数据规范化的DataMapping
  • 基于多个字段的高级对帐的LookupTable

数据映射

由库存脚本收集的原始数据有时需要进行规范化,然后才能导入iTop,以便获得同质的数据。该框架提供了用于执行简单规范化任务的帮助程序类MappingTable。

映射表(在params.xxx.xml配置文件中)配置为模式的有序列表,并且每个模式都关联有价值。映射表返回的“干净”价值是与匹配输入价值的第一个模式相关联的价值。模式表示为正则表达式。值可以使用占位符来引用匹配模式的某些部分(%1 $ s是整个模式,%2 $ s是常规价值中的第一组,等等)。

配置的示例(品牌标准化):

  <brand_mapping type="array">
    <!-- Syntax /pattern/replacement where:
      any delimiter can be used (not only /) 
      but the delimiter cannot be present in the "replacement" string
      pattern is a RegExpr pattern
      replacement is a sprintf string in which:
          %1$s will be replaced by the whole matched text,
          %2$s will be replaced by the first matched group, if any group is defined in the RegExpr
          %3$s will be replaced by the second matched group, etc...
    -->
    <pattern>/IBM/IBM</pattern>
    <pattern>/Hewlett.Packard/Hewlett-Packard</pattern>
    <pattern>/Dell/Dell</pattern>
    <pattern>/.*/%1$s</pattern>
  </brand_mapping>

此示例文件执行以下规范化:

  1. 每个包含“ IBM”的字符串都将转换为“ IBM”,
  2. 每个包含“ Hewlett”,后跟任何字符和“ Packard”的字符串都将转换为“ Hewlett-Packard”,
  3. 每个包含“ Dell”的字符串都将转换为“ Dell”,
  4. 所有其他字符串保持原样。

在代码中使用映射表

  1. 创建MappingTable类的实例,并将其传递给XML标签的名称,以在其中查找其配置(在XML param文件内部)
  2. 根据需要使用MapValue方法对每个价值进行流程处理(当在映射表中找不到匹配项时,第二个参数是默认的价值)。

用法示例:

// Turns the raw brand string ('brand_id') into a normalized brand // Use 'Other' for brands not found in the normalization table class TestCollector extends SQLCollector { protected $oBrandMapping; public function Prepare() { $bRet = parent::Prepare(); // Create the MappingTable once at the initialization of your collector $this->oBrandMapping = new MappingTable('brand_mapping'); return $bRet; } public function Fetch() { $aData = parent::Fetch(); if ($aData !== false) { // Then process each collected brand $aData['brand_id'] = $this->oBrandMapping->MapValue($aData['brand_id'], 'Other'); } return $aData; } }

高级查找

嵌入在iTop中的数据同步机制无法执行基于多个字段的协调(例如,基于品牌名称和模型名称搜索模型)。 LookupTable类为任何数量的字段提供此协调模型。

LookupTable类通过检索一组iTop对象的指定字段并将对象的结果标识符存储在iTo​​p中来构建查找表。

通过指定OQL查询(要检索的iTop对象的集合)和将用于映射的对象的字段来创建LookupTable的实例。

在创建LookupTable实例期间指定的字段列表(和顺序)是稍后执行Lookup(…)时要传递的字段列表。

初始化LookupTable后,对Lookup($ aData,array(Field1,Field2,…),desield)方法的调用将在$ aData中用iTop对象的标识符替换desield列的价值,其指定字段与值在$ aData中作为字段Field1,Field2…传递。

版本1.0.6开始,如果未找到相应的查找,则Lookup方法将返回false。在这种情况下,代码可以供应的默认价值抛出IgnoredRowException异常,以告知采集器拒绝所收集的数据的整个行。

版本1.1.4开始,LookupTable的构造函数接受一个额外的(可选)参数:$ bIgnoreMappingErrors(默认为false)。如果将此参数设置为true,则LookupTable将认为查找错误是正常的,并且不会报告将其作为警告(但仍在调试模式下列出)。当Lookuptable用于根据iTop中定义的目录过滤收集的数据时,这很有用。在这种情况下,查找错误是预期的行为。

在iTop中,运行中的系统版本表示为版本OS家族上的版本依赖。 iTop中可以包含以下对象:

  • Windows 7.0和8.1版,
  • Debian版本12.0.0。

它将存储在iTo​​p中,如下所示:

对象类id名称 
OSFamily1视窗 
OSFamily2Linux Debian 
对象类idosfamily_id名称
操作系统版本117.0.0
操作系统版本218.1.0
操作系统版本3212.0.0

现在,让我们想象一下,我们的采集器脚本为我们提供了两个信息:“ Windows”和“ 8.1.0”。我们可以将“ Windows”文本字符串存储在数据同步表的“ osfamily_id”字段中,并配置同步数据源以基于“名称”执行对帐(这将正确地将“ Windows”替换为1)。

但是要检索Windows的版本8.1.0的标识符(在我们的示例中为2),我们需要OS家族(“ Windows”)和版本号(“ 8.1.0”)。同步版本源无法执行此复合查找,而LookupTable才起作用。

$oOSVersionLookup = new LookupTable('SELECT OSVersion', ('osfamily_id_friendlyname', 'name'));

这将构建-在内存中-下表:

lookup_keyid
Windows_7.0.01
Windows_8.1.02
Debian_12.0.03

因此,如果我们在$ aData中具有以下值:

osfamily_idosversion_id
Windows8.1.0

致电:

$oOSVersionLookup->Lookup($aData, ('osfamily_id', 'osversion_id'), 'osversion_id', 0);

将查找结果$ aData ['osfamily_id']和$ aData ['osversion_id']的结果放入“ osversion_id”列。

$ aData将包含以下值:

osfamily_idosversion_id
Windows2

然后,我们必须配置同步数据源,以便它按原样接受oversion_id,而无需对其进行任何协调。

Lookup(...)的最后一个参数必须包含正在处理的CSV文件中的行号。在内部使用它仅在处理文件的第一行时执行一次初始化。

高级协调通过在数据集合之后但在将数据推到iTop之前(通过RESTTJSON API)检索要与组合键匹配的对象来进行。因此,为了使用此高级查找机制,必须告知框架采集器必须在实际数据之前重新处理收集的数据。这是通过重载采集器的MustProces/BeforeSynchro方法来实现的;然后返回true。

采集器框架提供了两个可以重载的其他方法:

  1. 在数据收集之后但在开始重新处理收集的数据的每一行之前,将调用InitProces/BeforeSynchro。这是在其中创建LookupTable实例的地方
  2. 对收集的数据的每一行(包括CSV文件的标题行,索引为零)调用ProcessLineBeforeSynchro。

使用范例

以下代码片段显示了使用表的全部情况:一种用于品牌+模型,一种用于OS家族+ OS版本。

protected function MustProcessBeforeSynchro() { // We must reprocess the CSV data obtained from the inventory script // to lookup the Brand/Model and OSFamily/OSVersion in iTop return true; } protected function InitProcessBeforeSynchro() { // Retrieve the identifiers of the OSVersion since we must do a lookup based on two fields: Family + Version // which is not supported by the iTop Data Synchro... so let's do the job of an ETL $this->oOSVersionLookup = new LookupTable('SELECT OSVersion', ('osfamily_id_friendlyname', 'name')); // Retrieve the identifiers of the Model since we must do a lookup based on two fields: Brand + Model // which is not supported by the iTop Data Synchro... so let's do the job of an ETL $this->oModelLookup = new LookupTable('SELECT Model', ('brand_id_friendlyname', 'name')); } protected function ProcessLineBeforeSynchro(&$aLineData, $iLineIndex) { // Process each line of the CSV if (!$this->oOSVersionLookup->Lookup($aLineData, ('osfamily_id', 'osversion_id'), 'osversion_id', $iLineIndex)) { throw New IgnoredRowException('Unknown OS Version'); } if (!$this->oModelLookup->Lookup($aLineData, ('brand_id', 'model_id'), 'model_id', $iLineIndex)) { throw New IgnoredRowException('Unknown Model'); } }

数据模型的变体

目标数据模型可能会发生一些变化(安装过程中选择的模块上的依赖)。如果某些配置中缺少给定的属性,则可以通过重载AttributeIsOptional方法来告诉采集器接受这种变化。 (这比为每种组合编写特定的采集器更为简单)。

如果缺少同步数据Source的JSON定义中指定的属性,除非属性声明为可选,否则处理将以错误停止。在后一种情况下,跳过的属性的名称记录在受保护的成员变量$ this-> aSkippedAttributes中,然后继续处理。属性的代码以后可以检查数组$ this-> aSkippedAttributes的内容,以确定是否必须收集哪些字段。

AttributeIsOptional的实施的示例作为VirtualMachineCollector类的方法:

public function AttributeIsOptional($sAttCode) { // If the module Service Management for Service Providers is selected during the setup // there is no "services_list" attribute on VirtualMachines. Let's safely ignore it. if ($sAttCode == 'services_list') return true; return parent::AttributeIsOptional($sAttCode); }

故障排除

对协调机制进行故障排除时,将库存脚本报告的原始(原始)值与协调流程的结果进行比较非常有用。只要采集器的MustProcessBeforeSynchro方法返回true,框架就会在数据子目录中生成两个文件。通过比较两个CSV文件,您可以轻松地在查找之前比较这些值:

  1. <collector_name>.raw-<index>.csv: :由库存脚本生成的原始数据,
  2. <collector_name>-<index>.csv: 重新处理的数据,将上传到iTop。

原贴链接:https://www.itophub.io/wiki/page?id=extensions%3Aitop-data-collector-base 


Data Collector Base

name:
Data Collector Base
description:
Inventory Data Collector toolkit for creating your own data collectors for iTop
version:
1.1.4
release:
2020-07-07
state:
stable
diffusion:
Client Store, Combodo Site

This module provides the base for creating an industrial data collection and synchronization application for iTop. Developers can rely on this module to perform all the heavy lifting related to the iTop data import and synchronization process in order to focus on the data collection.

Download 1.1.4

Features

  • Simple API to make the creation of a new collector fast and easy.

  • Automatic creation and update of Synchronization Data Sources in iTop, based on JSON definitions.

  • Support small variations to the target Data Model via explicitely “Optional” attributes

  • Basic validation of the CSV format compared to the expected fields in the synchro data source.

  • Data upload and synchronization by chunks of max_chunk_size (configurable).

  • Extensible mechanism for handling configuration parameters.

  • Extensible list of placeholders in the JSON definition.

  • Automatic management (via placeholders) of the contact to notify and the user to use for running the synchro.

  • Validation of the minimum version of PHP and the needed extensions (as specified by the collector).

  • Command line tool to produce the JSON definition from an existing Synchro Data Source in iTop.

  • Capability to run independently: the configuration of the data sources, the data collection and the data synchronization.

  • Configurable log level (for console output or syslog logging)

  • Simple framework for quickly creating SQL based collectors

Revision History

Release DateVersionComments
2020-06-241.1.4* The path to the configuration file can now be specified via the option --config_file on the command line.
* The location where to store the collected data is now a parameter in the configuration file: data_path.
* Better checking of Data Source definitions to catch missing reconciliation keys
* Option on the Lookuptable class to treat lookup errors as normal
2020-04-301.1.3* New CSV collector
* Configurable timestamp added in the logs
* New option for usage: –help
2019-11-071.1.2Fix “undefined constant TABLENAME_PATTERN”
2019-10-281.1.1Contains upgrades from both 1.0.13 and 1.1.0
* Reject invalid characters for database_table_name
2019-10-281.1.0Based on 1.0.9
* Added the specific class MySQLCollector which forces the DB connection to use UTF-8 characters
2019-10-281.0.13* LookupTables can now be non case sensitive (since MySQL is not)
* Prevent a warning in SQLCollector for each “ignored” attribute
* Improved support of iTop 2.4+ (obsolescence flag)
2019-10-281.0.12* removed a warning in PHP 7.2
2018-06-261.0.11Added a debug trace (visible if console_log_level=9) to show which mapping regular expression is applied (when one is applied).
Bug fix: properly handle utf-8 characters in the mapping table's regular expressions (/u modifier)
Make the cUrl/SSL options configurable to suit all possible combinations and security considerations.
2015-06-301.0.10New class of collector: MySQLCollector which forces the retrieved data to be encoded in UTF-8.
2015-06-091.0.9Performance enhancement: retrieve only the needed fields when building a lookup table.
2015-06-021.0.8Better checking of files access rights for writing. SQL connection string (for SQL collectors) is now fully configurable.
2015-05-201.0.7Bug fixes: Support of backslashes in file names. Removed a warning by marking Utils::Substitute() static.
2015-05-131.0.6Added the support of “ignoring” some rows in the data while re-processing them. SQL collector can be configured to safely ignore some fields.
2015-02-161.0.4Added the configuration parameter stop_on_synchro_error.
2015-01-061.0.3Handling of non UTF-8 data (via the overloading of GetCharset()), error checking for the data import phase, optimization for iTop 2.1.0: ignoring any change in the database_table_name field.
2014-11-031.0.2Added the base class SQLCollector for easily creating SQL based collectors.
2014-10-111.0.1Added the method AttributeIsOptional to handle variations in the target Data Model.
2014-05-131.0.0First version

Limitations

  • Data upload to iTop is done only via the syncho_import web service (could use the command line version or direct SQLcommands. TBD later, maybe)

  • Prior to the revision 3805 of iTop from SVN (from 2015-10-12!) the collector will NOT work properly if the account used to connect to iTop is not configured to use English as the language !!

Requirements

  • PHP Version 5.3.0 (support of namespaces may be required by some collectors)

  • An access to the iTop web services (REST + synchro_import.php and synchro_exec.php)

  • We recommand to install php_curl to use the collector base parameter itop_synchro_timeout otherwise the timeout is hardcoded to 200 secondes and can't be overwritten by the collector.

Under 1000 synchronized objects, it should work without php_curl, above it won't!

Installation

  • Expand the content of the zip archive on a folder on the machine were the collector will run.

  • Edit the content of the file conf/params.local.xml to suit your installation.

Configuration

params.local.xml is the only file to edit to configure a collector.

At minimum the following parameters must be set in this file:

  <itop_url>https://localhost/</itop_url>
  <itop_login>admin</itop_login>
  <itop_password>admin</itop_password>
ParameterMeaningSample value
itop_loginLogin (user account) for connecting to iTop. Must have admin rights for executing the data synchro.admin
itop_passwordPassword for the iTop account. 
itop_urlURL to the iTop Applicationhttps://localhost/itop

Optional parameters

The following parameters can be redefined to alter the default behavior of the collector:

ParameterMeaningDefault value
max_chunk_sizeMaximum number of elements to process in one iteration (for upload and synchro in iTop). If there are more elements than this number, the process will automatically iterate.1000
itop_synchro_timeoutTimeout for waiting for the execution of one data synchro task (in seconds)- requires php_curl600
stop_on_synchro_errorWhether or not to stop when an error occurs during a synchronization (yes or no).no
console_log_levelLevel of ouput to the console. From -1 (none) to 9 (debug).6 (info)
console_log_dateformatLogger timestamp format[Y-m-d H:i:s]
curl_optionsWhen using cUrl to connect to the iTop Webservices the cUrl options can be specified in this section. The syntax is <CURLOPT_NAME_OF_THE_OPTION1>VALUE_1</CURLOPT_NAME_OF_THE_OPTION1> where VALUE_x are either:
The numeric value of the option,
or the string representation of the corresponding PHP “define” (case sensitive).
It is possible to define several php_curl options like in the example below
 
data_pathNew in 1.1.4 The path where to store the temporary files generated by the collector. You can use the special placeholder %APPROOT% to specify a pth relative to the root folder of the collector.%APPROOT%/data
<curl_options>
    <CURLOPT_SSL_VERIFYHOST>0</CURLOPT_SSL_VERIFYHOST>
    <CURLOPT_SSL_VERIFYPEER>1</CURLOPT_SSL_VERIFYPEER>
  </curl_options>

The file params.distrib.xml contains the default values for the parameters. Both files (params.distrib.xml and params.local.xml) use exactly the same format. But params.distrib.xml is considered as the reference and should remain unmodified. Should you need to change the value of a parameter, copy and modify its definition in params.local.xml. The values inparams.local.xml have precedence over the ones in params.distrib.xml

Usage

To launch the data collection and synchronization with iTop, run the following command (from the root directory where the application is installed):

php exec.php

The following (optional) command line options are available:

OptionMeaningdefault value
--config_fileSpecify the full path to the configuration file. The file conf/params.local.xml is used by default if this parameter is omitted.empty
--console_log_level=<level>Level of ouput to the console. From -1 (none) to 9 (debug).6 (info)
--collect_onlyRun only the data collection, but do not synchronize the data with iTopfalse
--synchro_onlySynchronizes the data previously collected (stored in the data directory) with iTop. Do not run the collection.false
--configure_onlyCheck (and update if necessary) the synchronization data sources in iTop and exit. Do NOT run the collection or the synchronization 
--max_chunk_size=<size>Maximum number of items to process in one pass, for preserving the memory of the system. If there are more items to process, the application will iterate.1000
--helpUsage mode to display exec.php help. 

Running several instances of the collector

In many circumstances it may be useful to run several times the collector with a different set of parameters. For example to collect persons information from several LDAP servers (iTop Data Collector for LDAP) or Virtual Machines information from several vSphere servers (iTop Data Collector for vSphere).

Prior to version 1.1.4 of the framework, you had to completely duplicate the collector application and adjust the file conf/params.local.xml on each copy.

Since version 1.1.4 you can have just one single copy the of the collector application and specify a different configuration file (with the command line option --config_file) for each collection to run (i.e. one configuration file per LDAP or vSphere server).

However, to avoid any troubles during the collection of the data and the synchronization with iTop, the following parameters must be properly configured inside the configuration file:

  • Use a different <prefix> inside each different configuration file. This ensures that a specific set of Synchronization Data Sources will be created for each configuration file.

  • Use a different <data_path> variable for each configuration file. This will cause the collector to store all its collected data (including some temporary files) in a dedicated directory. This prevents one instance of the collector to overwrite the data of another one. You can use the syntax<data_path>%APPROOT%/data/collector1</data> to have a subfolder collector1 created inside the data folder.


Creating a collector

The specifics about a collector resides inside the “collectors” folder. There must be at least one file main.php inside this folder. The purpose of main.php is to register all the Collector classes for your module and load the corresponding classes (either via require_once(…) or by registering an auto-loader).

A collector is a PHP class that provides the data for a given Synchronization Data Source. Collector classes are derived from the abstract Collector class. Each collector is associated with a Synchronization Data Source, defined in JSON format. The default implementation simply looks for a JSON file with the same name as the collector class and the extension “.json”, in the collectors folder.

Specifying required extensions

If your collector needs a specific extension (or a minimum PHP version), you can indicate this dependency by calling the static method Orchestrator::AddRequirement($sMinRequiredVersion, $sExtension = 'PHP') in main.php:

For example:

Orchestrator::AddRequirement('5.4.0'); //This requires at least PHP 5.4
Orchestrator::AddRequirement('1.2.0', 'ldap'); //This requires at least the ldap extension version 1.2.0

Creating the JSON definition file

The simpler way to create the JSON file for a Synchro Data Source, is to export the definition of an existing data source.

  • Create the synchronisation data source in iTop, adjust its parameters (attributes, etc.) to suit your needs

  • Use the command line tool dump_tasks.php (available in the toolkit folder to produce the JSON file:

php toolkit/dump_tasks.php --task_name="name of the task to export" > collectors/myCollector.json

Inside your data source definition you can use special placeholders to make the data source configurable by the user of the application, or to adjust its behavior via some special settings:

Placeholder codeMeaningSample value
$version$The version of the module. Useful for versioning your application, for example in the “description” of the synchro data source.1.0.0
$synchro_user$The user to run the synchro, specified by its login in the configuration file. The identifier of the User object is available via this placeholder.12
$contact_to_notify$The contact to notify, specified by its email address in the configuration file. The identifer of the contact is supplied via this placeholder.48

On top of the 3 special configuration parameters listed above, all parameters defined in the section 'json_placeholders' of the configuration file are also available as placeholders inside the JSON file.

Sample configuration file:

params.local.xml
 
<?xml version="1.0" encoding="UTF-8"?>
<!--  Local values for parameters. -->
<!--  The values defined in this file have precedence over the ones defined in params.distrib.xml -->
<parameters>
  <itop_url>https://localhost/trunk</itop_url>
  <itop_login>admin</itop_login>
  <itop_password>admin</itop_password>
  <console_log_level>9</console_log_level>
  <contact_to_notify>denis.flaven@combodo.com</contact_to_notify>
  <synchro_user>admin</synchro_user>
  <json_placeholders type="hash">
    <test>Test 1</test>
  </json_placeholders>
</parameters>

Sample Synchro Data Source definition file, notice the use of the $version$, $synchro_user$, $contact_to_notify$ and $test$ placeholders:

If you specify a non-empty value for database_table_name, this name MUST BEGIN WITH<prefix>synchro_data. Where <prefix> is the prefix used for all tables in iTop (configured using the db_subname parameter in the iTop configuration file).

If you modify the list of fields of a class already collected, you must update your JSON definition file, to specify what to do with the new fields. By default they are added to existing Data Synchro with no update and no lock.

Implementing your collector

Your collector must be a class derived from Collector. It must implement (at least) the Fetch() method. Fetch must return either, for each object to load, an array using the format attribute_code => value or false when the end of the set of objects has been reached.

The array returned by Fetch() must contain:

  • an entry primary_key that uniquely identifies the object being synchronized with iTop. The entry can contain whatever unique ID you can obtain from the inventory collection, or a unique identifier generated as a combination of the various fields of the object. It's up to the collector application to guarantee the unicity of this identifier (and its stability in time)

  • an entry for each attribute of the object to be loaded in iTop.

The sample code below generates a set of 10 servers, named 'Server 1', 'Server 2' … 'Server 10', and initialized 3 fields of the servers: their name, their organization (always 'Demo') and their description.

main.php
 
class MyCollector extends Collector
{
  protected $idx;
 
  public function Prepare()
  {
        $bResult = parent::Prepare();
        $this->idx = 0;
        return $bResult;
  }
 
  public function Fetch()
  {
     if ($this->idx < 10)
     {
        $this->idx++;
        return (
           'primary_key' => $this->idx, 
           'name' => 'Server '.$this->idx, 
           'org_id' => 'Demo', 
           'description' => 'Test Collector'
        );
     }
     return false;
  }
}
 
// Register the collector, as the 1st to run
Orchestrator::AddCollector(1, 'MyCollector');

If the data returned by your collector are not encoded in the UTF-8 character set, overload the method GetCharset() of your collector to return the name of the character set (must return a value accepted by iconv on the iTop server)

Registering your collector

To register your collector, call the static method Orchestrator::AddCollector(). The two parameters are:

  • The order in which the collector should be run (when you need to run several collectors one after the other)

  • The name of the class (derived from Collector) in which the collector is implemented.

Default values for the parameters

A collector module can provide default values for its parameters by providing a file params.distrib.xml in the collectors folder. If such a file exists, its values are merged over the equivalent file in the conf directory.

Existing Base Collectors

To create a new collector, you can base it on the standard or use one of those recently added Collectors which already does part of the job depending on your data source:

  • CSV Collector

  • SQL Collector

  • JSON Collector (to be released in 2021)

SQL Collectors

The 'core' folder provides an abstract class SQLCollector which can serve as the basis for quickly creating collectors that retrieve their data via a SQL query.

To create such a collector you need to:

  1. Create a class derived from SQLCollector

  2. Create the json definition file for the data synchro source

  3. Add a configuration parameter (in params.distrib.xml) to define the SQL query to run

  4. Register your collector in collectors/main.php

The configuration parameters for the SQL Collectors are:

ParameterMeaningDefault Value
sql_engineThe PDO driver/engine to use for the database connection.mysql
sql_hostThe name or IP address of the database server to connect to.localhost
sql_databaseThe name of the database to connect to.empty
sql_loginThe login to use when connecting to the databaseroot
sql_passwordThe password to use when connecting to the databasen/a
sql_connection_stringNew in 1.0.8 The format of the PDO connection string. 3 placeholders are available inside the format string: %1$s = sql_engine, %2$s= sql_database and %3$s = sql_host%1$s:dbname=%2$s;host=%3$s
collector_class_queryThe query to run for the collector which PHP class is collector_class 
collector_class_ignored_attributesNew in 1.0.6 To take into account the possible variations of the data model, without re-writing a collector each time, it is possible to mark some of the collected attributes as “optional” so that the collector can run even if the corresponding attribute does not exist in the data model. Supply an array of attribute codes to ignore, here. 

To specify a port number (or any other driver specific options), change the format of thesql_connection_string. For example: %1$s:dbname=%2$s;host=%3$s;port=3307

For versions prior to 1.0.8, to specify a port number (other than the default port), use the syntax host;port=xxxx for the sql_host parameter. Example: localhost;port=3307

Starting with version 1.0.10, the framework provides a new class of collector: MySQLCollector. This class is identical to SQLCollector except that it forces the retrieved data to be encoded in UTF-8 by issuing the SQL command SET NAMES 'utf8' at the beginning of the each connection to the database. To avoid any problem with the character set of the data, it is recommended to use this new class for all connections to a MySQL/MariaDB database.

Example of a simple SQL Collector

Let's create a very simple SQL collector which copies the “Notes” documents (class DocumentNote) from one iTop instance to another. Since the collector inherits all its behavior from the base class, the PHP code for the collector is simply:

DocumentNotesCollector.class.inc.php
 

<?php class DocumentNoteCollector extends SQLCollector { }

Find here a sample of an SQL collector definition file.

Then in params.distrib.xml, add the following entries:

  <sql_database>test</sql_database>
  <sql_login>root</sql_login>
  <sql_password>s3cret</sql_password>
  <documentnotecollector_query>SELECT id as primary_key, name, text, description, status, '2.0' as version, documenttype_id, 1 as org_id FROM view_DocumentNote</documentnotecollector_query>
  <documentnotecollector_ignored_attributes type="array">
    <attribute>location_id</attribute>
    <attribute>version_id</attribute>
  </documentnotecollector_ignored_attributes>

Finally, in collectors/main.php add the following lines:

main.php
 
<?php
require_once(APPROOT.'collectors/DocumentNoteCollector.class.inc.php');
Orchestrator::AddCollector(1 /* $iRank */, 'DocumentNoteCollector');

CSV Collector

The 'core' folder provides an abstract class CSVCollector which can serve as the basis for quickly creating collectors that retrieve their data from CSV files.

To create such a collector you need to:

  1. Create a class derived from CSVCollector

  2. Create the json definition file for the data synchro source

  3. Add a configuration parameter (in params.distrib.xml) to define the CSV file to parse

  4. Register your collector in collectors/main.php

The configuration parameters for the CSV Collectors are:

ParameterMeaningDefault Value
collector_class_csvThe csv file to parse for the collector which PHP class is. You can specify the full path of this file (/tmp/myfile.csv) or a relative path to the collector collector_class. This parameter is mandatory 
collector_class_commandThe CLI command to execute BEFORE parsing csv file for the collector which PHP class is collector_class. This parameter is optional 
collector_class_encodingThe csv file encoding for the collector which PHP class is collector_class. This parameter is optionalUTF-8
collector_class_separatorThe separator to use for the csv file to parse. This parameter is optional;

Example of a simple CSV Collector

Let's create a very simple CSV collector which copies the “Person” objects (class Person)

Since the collector inherits all its behavior from the base class, the PHP code for the collector is simply:

iTopPersonCsvCollector.class.inc.php
 

<?php class iTopPersonCsvCollector extends CSVCollector { }

Find here a sample of Collector definition for CSV

Then in params.distrib.xml, add the following entries:

<?xml version="1.0" encoding="UTF-8"?>
<parameters>
  <iTopPersonCsvCollector_csv>collectors/iTopPersonCsvCollector.csv</iTopPersonCsvCollector_csv>
  <iTopPersonCsvCollector_command>sed -i -e 's|isaac|ISAAC|g' collectors/iTopPersonCsvCollector.csv</iTopPersonCsvCollector_command>
  <iTopPersonCsvCollector_encoding>UTF-8</iTopPersonCsvCollector_encoding>
</parameters>

Finally, in collectors/main.php add the following lines:

main.php
 
<?php
require_once(APPROOT.'collectors/iTopPersonCsvCollector.class.inc.php');
Orchestrator::AddCollector($index++, 'iTopPersonCsvCollector');

JSON Collector

The 'core' folder provides an abstract class JSONCollector which can serve as the basis for quickly creating collectors that retrieve their data from JSON files.

To create such a collector you need to:

  1. Create a class derived from JSONCollector

  2. Create the json definition file for the data synchro source

  3. Add a configuration parameter (in params.distrib.xml) to define the JSON file to parse

  4. Register your collector in collectors/main.php

The configuration parameters for the JSON Collectors are:

ParameterMeaning
collector_class_jsonfileDefine relative or absolute path to the json file to parse for the collector which PHP class is collector collector_class. This parameter orcollector_class_jsonurl is mandatory
collector_class_jsonurlThe URL of json file to parse for the collector which PHP class is. This parameter or collector_class_jsonpath is mandatory
collector_class_jsonpostXml of params to post with the url in order to get Json file <name_of_param>value</name_of_param>
collector_class_commandThe CLI command to execute BEFORE parsing json file for the collector which PHP class is collector_class. This parameter is optional
collector_class_wayway in order to find data to synchronize in json separator is / and * replace any word by example aa/bb for {“aa”:{“bb”:{mydata},“cc”:“xxx”} and aa/ * /bb for {“aa”:{cc“:{“bb”:{mydata1}},”dd“:{“bb”:{mydata2}}}
collector_class_fieldsxml witch describe connection between name in json and name in itop <name_in_json>name_in_itop</name_in_json>

Example of a simple JSON Collector

Let's create a very simple JSON collector which copies the “Person” objects (class Person) from one iTop to another

Since the collector inherits all its behavior from the base class, the PHP code for the collector is simply:

ITopPersonJsonCollector.class.inc.php
 

<?php class ITopPersonJsonCollector extends JsonCollector { }

Find here a sample for Collector definition file for JSON

Then in params.distrib.xml, add the following entries:

<?xml version="1.0" encoding="UTF-8"?>
<parameters>
        <itoppersonjsoncollector_jsonurl>http://localhost/iTop/webservices/rest.php</itoppersonjsoncollector_jsonurl>
        <itoppersonjsoncollector_jsonpost>
                <auth_user>restuser</auth_user>
                <auth_pwd>restuserpassword</auth_pwd>
                <json_data>{"operation": "core/get", "class": "Person", "key": "SELECT Person WHERE email LIKE '%.com'", "output_fields": "friendlyname, email, first_name, function, name, id, org_id,phone"}</json_data>
                <version>1.3</version>
        </itoppersonjsoncollector_jsonpost>
        <itoppersonjsoncollector_way>objects/*/fields</itoppersonjsoncollector_way>
        <itoppersonjsoncollector_fields>
                <primary_key>id</primary_key>
                <name>name</name>
                <status>status</status>
                <first_name>first_name</first_name>
                <email>email</email>
                <phone>phone</phone>
                <mobile_phone>mobile</mobile_phone>
                <function>function</function>
                <employee_number>employee_number</employee_number>
                <org_id>org_id</org_id>
        </itoppersonjsoncollector_fields>
        <itoppersonjsoncollector_defaults>
                <org_id>Demo</org_id>
                <status>active</status>
        </itoppersonjsoncollector_defaults>
        <json_placeholders>
                <!-- For compatibility with the version 1.1.x of the collector, define the data table names as following:
             <prefix></prefix>
             <persons_data_table>synchro_data_PersonAD</persons_data_table>
             <users_data_table></users_data_table>
              -->
                <prefix>$prefix$</prefix>
                <persons_data_table>synchro_data_person</persons_data_table>
        </json_placeholders>
</parameters>

Finally, in collectors/main.php add the following lines:

main.php
 
<?php
require_once(APPROOT.'collectors/ITopPersonJsonCollector.class.inc.php');
Orchestrator::AddCollector($index++, 'ITopPersonJsonCollector');

Advanced collectors

The collector framework provides means to perform some advanced processing for real-life collectors:

  • DataMapping for configurable data normalizations

  • LookupTable for advanced reconciliations based on multiple fields

Data Mapping

Raw data collected by inventory scripts sometimes require a normalization before being imported into iTop, in order to obtain homogenous data. The framework provides the helper class MappingTable for performing simple normalizations tasks.

A mapping table is configured (in the params.xxx.xml configuration file) as an ordered list of patterns, with a value associated to each pattern. The “clean” value returned by the mapping table is the value associated with the first pattern that matches the input value. Patterns are expressed as regular expressions. Values can use the placeholders to refer to some part of the matched pattern (%1$s is the whole pattern, %2$s the first group inside the regular expression, etc.).

Example of configuration (Brand normalization):

  <brand_mapping type="array">
    <!-- Syntax /pattern/replacement where:
      any delimiter can be used (not only /) 
      but the delimiter cannot be present in the "replacement" string
      pattern is a RegExpr pattern
      replacement is a sprintf string in which:
          %1$s will be replaced by the whole matched text,
          %2$s will be replaced by the first matched group, if any group is defined in the RegExpr
          %3$s will be replaced by the second matched group, etc...
    -->
    <pattern>/IBM/IBM</pattern>
    <pattern>/Hewlett.Packard/Hewlett-Packard</pattern>
    <pattern>/Dell/Dell</pattern>
    <pattern>/.*/%1$s</pattern>
  </brand_mapping>

This example file performs the following normalization:

  1. Every string containing “IBM” is transformed into “IBM”,

  2. Every string containing “Hewlett”, followed by any character, followed by “Packard” is transformed into “Hewlett-Packard”,

  3. Every string containing “Dell” is transformed into “Dell”,

  4. All other strings are kept as is.

Using a mapping table in your code

  1. Create an instance of the MappingTable class, passing it the name of the XML tag in which to look for its configuration (inside the XML param file)

  2. Use the MapValue method to process each value as needed (the second parameter is the default value, when no match is found in the mapping table).

Usage example:

// Turns the raw brand string ('brand_id') into a normalized brand // Use 'Other' for brands not found in the normalization table class TestCollector extends SQLCollector { protected $oBrandMapping; public function Prepare() { $bRet = parent::Prepare(); // Create the MappingTable once at the initialization of your collector $this->oBrandMapping = new MappingTable('brand_mapping'); return $bRet; } public function Fetch() { $aData = parent::Fetch(); if ($aData !== false) { // Then process each collected brand $aData['brand_id'] = $this->oBrandMapping->MapValue($aData['brand_id'], 'Other'); } return $aData; } }

Advanced Lookups

The data synchronization mechanism embedded in iTop is not capable of performing reconciliations based on multiple fields (like searching for a Model based on both the Brand name and the Model name). The LookupTable class provides this reconciliation capability for any number of fields.

The class LookupTable builds a lookup table by retrieving the specified fields of a set of iTop objects, and storing the resulting identifier of the objects in iTop.

An instance of LookupTable is created by specifying an OQL query (the set of iTop objects to retrieve) and the fields of the objects that will be used for the mapping.

The list (and the order) of the fields specified during the creation of the LookupTable instance is the list of fields to be passed later on when performing a Lookup(…)

Once the LookupTable has been initialized, a call to the Lookup($aData, array(Field1, Field2, …), destField) method will replace in $aData the value of the column destField by identifier of the iTop object whose specified fields match the values passed in $aData as the columns Field1, Field2….

Since version 1.0.6, the Lookup method returns false if not corresponding lookup was found. In such a case the code can either supply a default value, of throw an exceptionIgnoredRowException to tell the collector to reject the whole line of collected data.

Since version 1.1.4, the constructor of LookupTable accepts one extra (optional) parameter: $bIgnoreMappingErrors (default to false). If this parameter is set to true, the LookupTable will consider that lookup errors are normal and will not report them as warnings (but still list them in debug mode). This can be useful when the Lookuptable is used for filtering the collected data against a catalog defined in iTop. In such a case, lookup errors are the expected behavior.

Example

In iTop, the operating system version is represented as a version depending on an OS family object. We can have the following objects in iTop:

  • Windows, versions 7.0 and 8.1,

  • Debian version 12.0.0.

This will be stored in iTop as shown below:

Object classidname
OSFamily1Windows
OSFamily2Linux Debian
Object classidosfamily_idname
OSVersion117.0.0
OSVersion218.1.0
OSVersion3212.0.0

Now let's imagine that our collector script gives us the two informations: 'Windows' and '8.1.0'. We can store the 'Windows' text string in the 'osfamily_id' field of the data synchro table and configure the synchro data source to perform the reconciliation based on the 'name' (this will properly replace 'Windows' by 1).

But to retrieve the identifier of the version 8.1.0 of Windows (which is 2 in our example) we need both the OS Family ('Windows') and the version number ('8.1.0'). The Synchronization Data Source is not capable of doing this composite lookup, this where the LookupTable comes into play.

$oOSVersionLookup = new LookupTable('SELECT OSVersion', ('osfamily_id_friendlyname', 'name'));

This will build - in memory - the following table:

lookup_keyid
Windows_7.0.01
Windows_8.1.02
Debian_12.0.03

So if we have in $aData the following values:

osfamily_idosversion_id
Windows8.1.0

Calling:

$oOSVersionLookup->Lookup($aData, ('osfamily_id', 'osversion_id'), 'osversion_id', 0);

Will place in the column 'osversion_id' the result of the lookup for the values $aData['osfamily_id'] and $aData['osversion_id'].

$aData will then contain the following values:

osfamily_idosversion_id
Windows2

We then have to configure the Synchro Data Source so that it accepts the oversion_id as-is without performing any reconciliation on it.

The last parameter of Lookup(…) must contain the line number inside the CSV file being processed. This is used internally to perform some initializations only once when processing the first line of the file.

The advanced reconciliation works by retrieving (via the REST/JSON API), the objects to be matched against the composite key, after the data collection but before pushing the data to iTop. Therefore, in order to use this advanced lookup mechanism, you must tell the framework that the collector has to reprocess the collected data before the actual synchro. This is achieved by overloading the method MustProcessBeforeSynchro of the collector; and returning true.

The collector framework provides two additional methods which can be overloaded:

  1. InitProcessBeforeSynchro is called after the data collection, but before starting to reprocess each line of the collected data. This is the plece where to create the LookupTable instance

  2. ProcessLineBeforeSynchro is called for each line of the collected data (including the header line of the CSV file, which index is zero)

Usage Example

The following code fragment shows to use cases of lookup tables altogether: one for brand + model and one for OS family + OS version.

protected function MustProcessBeforeSynchro() { // We must reprocess the CSV data obtained from the inventory script // to lookup the Brand/Model and OSFamily/OSVersion in iTop return true; } protected function InitProcessBeforeSynchro() { // Retrieve the identifiers of the OSVersion since we must do a lookup based on two fields: Family + Version // which is not supported by the iTop Data Synchro... so let's do the job of an ETL $this->oOSVersionLookup = new LookupTable('SELECT OSVersion', ('osfamily_id_friendlyname', 'name')); // Retrieve the identifiers of the Model since we must do a lookup based on two fields: Brand + Model // which is not supported by the iTop Data Synchro... so let's do the job of an ETL $this->oModelLookup = new LookupTable('SELECT Model', ('brand_id_friendlyname', 'name')); } protected function ProcessLineBeforeSynchro(&$aLineData, $iLineIndex) { // Process each line of the CSV if (!$this->oOSVersionLookup->Lookup($aLineData, ('osfamily_id', 'osversion_id'), 'osversion_id', $iLineIndex)) { throw New IgnoredRowException('Unknown OS Version'); } if (!$this->oModelLookup->Lookup($aLineData, ('brand_id', 'model_id'), 'model_id', $iLineIndex)) { throw New IgnoredRowException('Unknown Model'); } }

Data Model Variants

It may happen that the target Data Model has some variants (depending on the set of modules chosen during the installation). If a given attribute can be missing in some configurations, you can tell your collector to accept this variation, by overloading the method AttributeIsOptional. (This is simpler than writing a specific collector for each combination).

If an attribute specified in the JSON definition of the Synchro Data Source is missing, the processing will stop with an error, unless this attribute is declared as optional. In the later case, the name of the skipped attribute is recorded in the protected member variable $this->aSkippedAttributes and the processing continues. The code of the collector can later check the content of the array $this->aSkippedAttributes to determine which fields have to be collected or not.

Example of implementation of AttributeIsOptional as a method of the VirtualMachineCollector class:

public function AttributeIsOptional($sAttCode) { // If the module Service Management for Service Providers is selected during the setup // there is no "services_list" attribute on VirtualMachines. Let's safely ignore it. if ($sAttCode == 'services_list') return true; return parent::AttributeIsOptional($sAttCode); }

Troubleshooting

When troubleshooting the reconciliation mechanism it is useful to compare the original (raw) values as reported by the inventory script with the result of the reconciliation process. Whenever the method MustProcessBeforeSynchro of a collector returns true, the framework generates two files inthe data subdirectory. You can easily compare the values before/after the lookup by comparing the two CSV files:

  1. <collector_name>.raw-<index>.csv: the original data, as produced by the inventory script,

  2. <collector_name>-<index>.csv: the reprocessed data, to be uploaded to iTop.

标签:
由 superadmin 在 2020/08/27, 16:05 创建
    

需要帮助?

如果您需要有关XWiki的帮助,可以联系:

深圳市艾拓先锋企业管理咨询有限公司