数据同步概述

联合多个来源

强大的数据同步引擎允许将多种信息源(例如,在使用网络发现工具或资产管理工具时)联合到iTop中。

使用数据源对象联合

在iTop中使用数据源对象定义了该联合。请参阅数据源 每个数据来源都定义了iTop如何处理来自给定来源的给定类型对象的同步。

用于同步数据的通用流程

用于将数据与iTop同步的通用(正在进行的)流程基于以下步骤:

  1. 外部源应用程序中提取数据[iTop未处理]
  2. 数据转换为适合iTop的内容格式[iTop不处理]
  3. 使用MySQL命令[未由iTop处理]将导入和数据放入iTop中的副本表中
  4. 索用于在iTop中匹配对象[由iTop处理]
  5. iTop中创建或更新同步对象[由iTop处理]
  6. iTop中显示和管理同步的对象

流程以两种可能的方式执行:

1:数据提要直接写入数据库

下面是一个示例,它介绍了一个XTL125501从外部来源到iTop的信息,它使用ETL直接在MySQL中编写流动。

Data synchro by writing into the SQL replica table

在iTop中实现数据同步

要在iTop中实现数据同步,请执行以下步骤:

  1. 在iTop管理员菜单中,为要导入的对象类型定义数据源对象。这将在iTop数据库中创建一个特定的副本表,名为“synchro_data_xx”。 (一旦创建了数据源,请检查数据源的“属性”选项卡。临时表的名称显示在顶部。)
  2. 使用常用ETL或普通的旧脚本,使用来自外部源的数据填充与数据源相对应的“synchro_data_xxxx”表。您可以使用特殊列primary_key将对象的任何标识符存储在外部应用中。该表中的每个数据将对应于iTop中的一个对象。列ID保留供iTop使用,无法写入。所有其他列对应于iTop对象的字段。
  3. 触发器通过执行页面“ synchroo”在iTop对象和临时表的副本之间执行数据synchro/synchro_exec.php”(无论是在命令行模式下还是通过调用网页(例如使用wget))。

当然,您可以定期(每小时,每天或每周)安排上述第2步和第3步。

有关更多信息,请参见副本。

2:数据通过CSV文件供稿

一种替代方法包括创建一个中间CSV文件,并使用synchro_import.php(由iTop提供)来将导入执行到MySQL中。

在这种情况下,信息流动如下所示:

Data synchro by writing a temporary file

  1. 外部源应用程序中提取数据[iTop未处理]
  2. 数据转换为适合iTop的内容格式[iTop不处理]
  3. 数据写入中间CSV文件中[iTop不处理]
  4. 入将数据放入iTop中的副本表中[由iTop处理]
  5. 索用于在iTop中匹配对象[由iTop处理]
  6. iTop中创建或更新同步对象[由iTop处理]
  7. iTop中显示和管理同步的对象

替代方法是使用服务 synchro_import.php  将CSV文件中的数据转换为导入以及将iTC中的动态转换为iTC即可。

CSV导入和数据同步有什么区别?

数据同步

数据同步是指以循环方式将导入数据从另一个系统转换为iTop。它可以从命令行或从Web运行,但不能交互运行。数据同步针对不经常使用数据的大量数据进行了优化。例如,每天可以在iTop中同步来自LDAP数据的10,000个联系人。每天不会修改多于一小部分的用户记录。 iTop可以有效地处理此问题。

同步数据时,iTop会跟踪iTop对象和数据的源之间的关系。因此,可以防止用户修改iTop中的同步对象(部分或全部),并通知他们数据的来源。这对于在iTop中“联合”数据的多个源很有用。

CSV导入

CSV导入可以交互方式运行,也可以从命令行运行。它更针对“一次性”进口。可以从脚本(使用命令行界面或从Web)或交互式使用它。执行CSV导入时,iTop不会提供有关记录来源的导入信息。将记录加载到iTop中后,授权用户可以修改对象,而原始源无需任何引用。

总结一下:

CSV导入适用于:

  • 将初始数据导入iTop
  • 在数据上执行批量转换(有时导出>在Excel中修改> re-import比直接在iTop中编辑对象要容易)

同步有益于:

  • 在iTop中的不同系统之间联合数据
  • 通过某些计划机制导入数据
  • 防止用户修改导入的数据

同步配置项目

当配置项与数据源同步时,iTop中的显示会略有不同。配置项上的操作活动也可能是受限的,数据源的配置上是依赖。

例如,对于最终用户,同步的对象的字段可能显示为只读。还可以配置iTop以防止用户删除同步的对象。同步对象的确切行为由每个配置项源的属性确定。

下面的屏幕快照显示了同步的服务器在iTop中的外观。 (请注意对象标题中的“锁定”图标及其附带的工具提示)。

https://www.itophub.io/wiki/media?w=400&tok=300edc&media=mediawiki%3Asynchronizedobject.png

增加了复杂性

设计

在设计此机制时,请牢记以下约束:

  • 为每天同步大量数据进行了优化
  • 与第三方数据管理工具(如Talend)轻松集成
  • 确保安全并易于操作故障排除

设置和常见陷阱

由于这些要求,设置起来并不像人们想象的那样直观。常见陷阱如下:

从同步数据表中删除记录

从同步数据表中删除记录不会删除对象。它将只是从同步机制中分离对象。用户将有权编辑或删除对象,就像在UI中创建对象一样。将对象重新连接到同步需要同时设置数据源,以便可以进行对帐,然后将记录重新引入到同步数据表中。请参阅上面的删除操作原理。同步将从您的同步数据表中删除记录。

定期重新创建完全同步数据表

通过相关的设置,这将起作用。但是,当表清空后(直到再次填充表且同步重复运行),用户将看不到相应的对象已同步。同步需要很长时间才能执行,并且将在iTop Web服务器上大量占用影响度和响应时间。对于同步的每个项目,都必须进行对帐,要求执行对搜索的查询才能获得最佳匹配。

使用此机制进行实时同步

通过诸如RESTTJSON之类的Web服务来实现它要容易得多。这样,该范例就更加直观:创建一个对象(分别更新并删除),然后您必须调用REST运维“对象创建”(分别更新并删除)。

原贴链接:https://www.itophub.io/wiki/page?id=2_7_0%3Aadvancedtopics%3Adata_synchro_overview


Data Synchronization Overview

Federating multiple sources

A powerful Data Synchronization engine allows multiple sources of information (for example when using a network discovery tool or an asset management tool) to be federated into iTop.

Federating using Data Source Objects

This federation is defined in iTop using Data Source ObjectsSee data source Each data source defines how iTop handles the synchronization of a given type of object coming from a given source.

The generic process for synchronizing data

The generic (on-going) process for synchronizing data with iTop is based on the following steps:

  1. Extract data from the external source/application [not handled by iTop]

  2. Transform the data to a content/format suitable for iTop [not handled by iTop]

  3. Import the data into a replica table in iTop, using MySQL commands [not handled by iTop]

  4. Search for matching objects in iTop [handled by iTop]

  5. Create/Update or Delete the synchronized objects in iTop [handled by iTop]

  6. Display and manage the synchronized object in iTop

This process is carried out in 2 possible ways :

1: Data feed writing directly to the database

Below is an example of the information flow from an external source into iTop, using an ETL to write the data directly in MySQL.

Data synchro by writing into the SQL replica table

Implement Data Synchronization in iTop

To implement Data Synchronization in iTop, follow the steps below:

  1. From the iTop admin menu, define a Data Source object, for the type of objects you want to import/synchronize. This creates a specific replica table in the iTop database, named “synchro_data_xxx”. (Once the data source is created, check the Attributes tab of the data source. The name of the temporary table is displayed at the top.)

  2. Using either your favorite ETL or plain old scripts, populate the “synchro_data_xxxx” table corresponding to the Data Source with the data coming from the external source. You can use the special column primary_key for storing any identifier of the object in the external application. Each record in this table will correspond to one object in iTop. The column id is reserved for use by iTop and cannot be written. All other columns correspond to fields of the iTop object.

  3. Trigger the execution of the data synchronization between the iTop objects and the replicas of the temporary table by executing the page “synchro/synchro_exec.php” either in command line mode, or by invocating the web page (using wget for example).

Of course, you can schedule steps 2 and 3 above on a regular basis (hourly, daily or weekly).

See Replicas for more information.

2: Data feed via a CSV file

An alternative method consists of creating an intermediate CSV file and using the php script synchro_import.php (provided by iTop) to perform the import into MySQL.

In this case, the information flow is as shown below:

Data synchro by writing a temporary file

  1. Extract data from the external source/application [not handled by iTop]

  2. Transform the data to a content/format suitable for iTop [not handled by iTop]

  3. Write the data into an intermediate CSV file [not handled by iTop]

  4. Import the data into the replica table in iTop [handled by iTop]

  5. Search for matching objects in iTop [handled by iTop]

  6. Create/Update or Delete the synchronized objects in iTop [handled by iTop]

  7. Display and manage the synchronized object in iTop

An alternative is to use the service synchro_import.php to import the data from a CSV file and synchronize it in iTop in a single step.

What is the difference between CSV Import and Data Synchronization?

Data Synchronization

Data Synchronization is meant to import data into iTop from another system, in a recurring manner. It can be run from the command line or from the web, but not interactively. Data Synchronization is optimized for large volumes of data that does not change often. For example, 10,000 contacts from an LDAP server, can be synchronized in iTop once a day. It is unlikely that more than a small percentage of the users' records will be modified on a daily basis. This is handled efficiently by iTop.

When synchronizing data, iTop keeps track of the relation between the iTop object and the source of the data. Therefore, it is possible to prevent users from modifying the synchronized objects (partially or totally) in iTop and to inform them where the data comes from. This is useful for “federating” several sources of data in iTop.

CSV import

The CSV import can be run interactively or from the command line. It is more targeted towards “one shot” imports. It can be used from a script (using the command line interface or from the web) or interactively. When performing a CSV import, iTop does not record information about the source of the data. Once the data has been loaded into iTop, the objects can be modified by the authorized users, without any reference to the original source.

To summarize:

CSV import is good for:

  • importing initial data into iTop

  • performing bulk transformations on the data (sometimes it is easier to export > modify in Excel > re-import, than to edit the objects directly in iTop)

Synchronization is good for:

  • federating data between different systems in iTop

  • importing data via some scheduled mechanism

  • preventing users from modifying the imported data

Synchronized Configuration Items

When a configuration item is synchronized with a data source, the display is slightly different in iTop. The actions on a CI may also be limited, depending on the configuration of the data source.

For example, the fields of the object that are synchronized may appear as read-only for the end users. It is also possible to configure iTop to prevent users from deleting a synchronized object. The exact behaviour of the synchronized CI is determined by the properties of each data source.

The screenshot below shows what a synchronized Server looks like in iTop. (Notice the “lock” icon in the object's title and the tooltip attached to it).

https://www.itophub.io/wiki/media?w=400&tok=300edc&media=mediawiki%3Asynchronizedobject.png

Added complexity

Design

This mechanism has been designed bearing in mind the following constraints:

  • to be optimized for synchronizing huge volumes of data on a daily basis

  • to integrate easily with third-party data management tools such as Talend

  • to be secured and easily operated/troubleshooted

Setting-up and common pitfalls

As a consequence of these requirements, setting it up is not as intuitive as one might imagine. Common pitfalls are described below:

Deleting records from the sync data table

Deleting records from the sync data table will not delete the objects. It will just detach the objects from the synchronization mechanism. Users will have the right to edit or delete the objects exactly as if they were created from within the UI. Re-attaching the objects to the synchronization requires both to setup the data source so that the reconciliation can occur, then re-introduce the records into the sync data table. See how the deletion works hereabove. The synchronization will delete the records from the synch data table for you.

Recreating the full sync data table periodically

With the relevant setup, this will work. However, when the table gets emptied, (and until it gets populated again AND the synchro is run repeated), users will NOT see the corresponding objects as being synchronized. Synchronization takes a long time to execute and it will heavily impact the response time on the iTop web server. For every item synchronized, a reconciliation will have to occur, requiring the execution of queries to search for the best match.

Using this mechanism for a realtime synchronization

It is far easier to implement this by the means of web services such as REST/JSON. The paradigm is then more intuitive: an object is created (respectively updated/deleted) then you have to invoke the REST operation “object create” (respectively update/delete).

标签:
由 superadmin 在 2020/08/27, 16:21 创建
    

需要帮助?

如果您需要有关XWiki的帮助,可以联系:

深圳市艾拓先锋企业管理咨询有限公司